메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


버젼차이로 인한 프로토콜이 맞지 않아서 서로 엉뚱하게 보내고 엉뚱하게 처리하고 있다.
0.9에서 0.10으로 넘어가면서 api명칭및 패키지 구조, 특히 Properties지정하는 항목이 변경되거나 필수항목이 추가되는 등의 많은 변화가 발생했다.

api는 
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
등을 0.9버젼에서 사용했는데... 0.10이상에서는 deplecated되고 org.apache로 시작하는 패키지를 사용한다.

-------------producer쪽 로그-----------------
16/08/18 20:00:17 INFO VerifiableProperties: Verifying properties
16/08/18 20:00:17 INFO VerifiableProperties: Property metadata.broker.list is overridden to sda1:9092,sda2:9092,sda3:9092
16/08/18 20:00:17 INFO VerifiableProperties: Property partitioner.class is overridden to kafka.producer.DefaultPartitioner
16/08/18 20:00:17 INFO VerifiableProperties: Property request.required.acks is overridden to 1
16/08/18 20:00:17 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.DefaultEncoder
Send start (OneM2M)......................
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,sda2,9092) with correlation id 0 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda2:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda2:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M -> 
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException 
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 1 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M -> 
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException 
16/08/18 20:00:18 ERROR DefaultEventHandler: Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: COL_ONEM2M
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 3
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(0,sda1,9092) with correlation id 2 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda1:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda1:9092
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 4 to broker 3 with data for partitions [COL_ONEM2M,0]
java.nio.BufferUnderflowException
        at java.nio.Buffer.nextGetIndex(Buffer.java:506)
        at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
        at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
        at kafka.producer.Producer.send(Producer.scala:78)
        at kafka.javaapi.producer.Producer.send(Producer.scala:35)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 2
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 5 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M -> 
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException 
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 7 to broker 2 with data for partitions [COL_ONEM2M,2]
java.nio.BufferUnderflowException
        at java.nio.Buffer.nextGetIndex(Buffer.java:506)
        at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
        at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
        at kafka.producer.Producer.send(Producer.scala:78)
        at kafka.javaapi.producer.Producer.send(Producer.scala:35)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 1
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,sda2,9092) with correlation id 8 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda2:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda2:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M -> 
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException 
16/08/18 20:00:18 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:00:18 WARN DefaultEventHandler: Failed to send producer request with correlation id 10 to broker 1 with data for partitions [COL_ONEM2M,1]
java.nio.BufferUnderflowException
        at java.nio.Buffer.nextGetIndex(Buffer.java:506)
        at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:40)
        at kafka.api.ProducerResponse$$anonfun$1$$anonfun$apply$1.apply(ProducerResponse.scala:36)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:36)
        at kafka.api.ProducerResponse$$anonfun$1.apply(ProducerResponse.scala:33)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
        at scala.collection.immutable.Range.foreach(Range.scala:160)
        at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:33)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
        at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:275)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:113)
        at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:105)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:105)
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
        at kafka.producer.Producer.send(Producer.scala:78)
        at kafka.javaapi.producer.Producer.send(Producer.scala:35)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/18 20:00:18 INFO DefaultEventHandler: Back off for 100 ms before retrying send. Remaining retries = 0
16/08/18 20:00:18 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,sda3,9092) with correlation id 11 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:00:18 INFO SyncProducer: Connected to sda3:9092 for producing
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from sda3:9092
16/08/18 20:00:18 WARN BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic COL_ONEM2M -> 
No partition metadata for topic COL_ONEM2M due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException}] for topic [COL_ONEM2M]: class org.apache.kafka.common.errors.UnknownTopicOrPartitionException 
16/08/18 20:00:18 ERROR DefaultEventHandler: Failed to send requests for topics COL_ONEM2M with correlation ids in [0,11]
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
        at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:96)
        at kafka.producer.Producer.send(Producer.scala:78)
        at kafka.javaapi.producer.Producer.send(Producer.scala:35)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.send(AvroOneM2MEmitter.java:59)
        at com.gooper.icbms.sda.kafka.onem2m.AvroOneM2MEmitter.main(AvroOneM2MEmitter.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Send end(OneM2M)......................
16/08/18 20:00:18 INFO Producer: Shutting down producer
16/08/18 20:00:18 INFO ProducerPool: Closing all sync producers
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:00:18 INFO Producer: Producer shutdown completed in 7 ms


----------------------------------consumer로그-----------------
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1614 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1615 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:10 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:10 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:10 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:10 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1616 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:10 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:10 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:10 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:10 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1617 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1618 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1619 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1620 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:11 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:11 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:11 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:11 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1621 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:11 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:11 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:11 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:11 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1622 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1623 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1624 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,localhost,9092) with correlation id 1625 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:12 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:12 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:12 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:12 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(3,localhost,9092) with correlation id 1626 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:12 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:12 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:12 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:12 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
^C16/08/18 20:04:13 INFO VerifiableProperties: Verifying properties
16/08/18 20:04:13 INFO VerifiableProperties: Property client.id is overridden to testgroup_11
16/08/18 20:04:13 INFO VerifiableProperties: Property metadata.broker.list is overridden to localhost:9092,localhost:9092,localhost:9092
16/08/18 20:04:13 INFO VerifiableProperties: Property request.timeout.ms is overridden to 30000
16/08/18 20:04:13 INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,localhost,9092) with correlation id 1627 for 1 topic(s) Set(COL_ONEM2M)
16/08/18 20:04:13 INFO SyncProducer: Connected to localhost:9092 for producing
16/08/18 20:04:13 INFO SyncProducer: Disconnecting from localhost:9092
16/08/18 20:04:13 INFO ConsumerFetcherManager: [ConsumerFetcherManager-1471517919733] Added fetcher for partitions ArrayBuffer([[COL_ONEM2M,2], initOffset 30 to broker BrokerEndPoint(2,localhost,9092)] , [[COL_ONEM2M,0], initOffset 42 to broker BrokerEndPoint(3,localhost,9092)] )
16/08/18 20:04:13 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-3], Error for partition [COL_ONEM2M,0] to broker 3:kafka.common.NotLeaderForPartitionException
16/08/18 20:04:13 ERROR ConsumerFetcherThread: [ConsumerFetcherThread-testgroup_11_sda1-1471517919659-fec46942-0-2], Error for partition [COL_ONEM2M,2] to broker 2:kafka.common.NotLeaderForPartitionException
번호 제목 글쓴이 날짜 조회 수
441 VisualVM 1.3.9을 이용한 JVM 모니터링 file 총관리자 2016.10.27 332
440 Cloudera가 사용하는 서비스별 포트 총관리자 2018.03.29 326
439 Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 TaskAttempt killed because it ran on unusable node 오류시 조치방법 총관리자 2017.04.06 324
438 [JSON 파싱]mongodb의 document를 GSON을 이용하여 parsing할때 ObjectId값에서 오류 발생시 조치방법 총관리자 2017.01.18 323
437 cloudera-scm-agent 설정파일 위치및 재시작 명령문 총관리자 2018.03.29 322
436 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable원인 총관리자 2015.04.27 322
435 [tomcat] logrotate를 이용하여 catalina.out로그파일 일별로 로테이션 저장하기 file 총관리자 2017.01.18 318
434 [Kudu] tablet server 혹은 kudu master가 어떤 원인에 의해서 replica가 failed상태인 경우 복구하는 방법 총관리자 2021.05.24 315
433 editLog의 문제로 발생하는 journalnode 기동 오류 발생시 조치사항 총관리자 2017.09.14 313
432 [Oozie]Disk I/O error: Failed to open HDFS file dhfs://..../tb_aaa/....OPYING 총관리자 2019.02.15 312
431 Cloudera설치중 실패로 여러번 설치하는 과정에 "Running in non-interactive mode, and data appears to exist in Storage Directory /dfs/nn. Not formatting." 오류가 발생시 조치하는 방법 총관리자 2018.03.29 309
430 cassandra cluster 문제가 있는 node제거 하기(DN상태의 노드가 있으면 cassandra cluster 전체에 문제가 발생하므로 반드시 제거할것) 총관리자 2017.06.21 309
429 Runtime.getRuntime().exec(cmd) sample 소스 총관리자 2015.11.19 305
428 spark-submit으로 spark application실행하는 다양한 방법 총관리자 2016.05.25 303
427 Spark에서 KafkaUtils.createStream()를 이용하여 이용하여 kafka topic에 접근하여 객채로 저장된 값을 가져오고 처리하는 예제 소스 총관리자 2017.04.26 292
» kafka 0.9.0.1버젼의 producer와 kafka버젼이 0.10.0.1인 consumer가 서로 대화하는 모습 총관리자 2016.08.18 288
425 룰에 매칭되면 발생되는 엑티베이션 객체에 대한 작업(이전값 혹은 현재값)을 처리하는 클래스 파일 총관리자 2016.07.21 285
424 System Properties Comparison Elasticsearch vs. Hive vs. Jena file 총관리자 2016.03.10 285
423 Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging 발생시 조치사항 총관리자 2017.04.19 284
422 여러 홈페이지를 운영하거나 혹은 서버에 가입한 사용자들에게 홈페이지 계정을 나누어 줄수 있도록 설정/계정 생성방법 총관리자 2018.01.23 282

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로