메뉴 건너뛰기

검색창 닫기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


hive java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from...오류해결방법

총관리자 2015.06.16 09:31 조회 수 : 1735 

Hive가 "GROUP BY"에서 최적화하면서 내부적으로 컬럼을 추가하여 reducer로 보내는데 이 컬럼은 실존하지 않는 컬럼으로 문제가 발생할 수 있음
hive 0.14.0에서 default가 disabled되었는데 이전 버전에서는 default가 enabled로 되어 있어서 문제가 발생하면
set hive.optimize.sort.dynamic.partition=false;
를 추가하여 준다.

-----오류내용----
2015-06-15 14:12:01,345 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1434344795088_0005_r_000000 Task Transitioned from SCHEDULED to RUNNING
2015-06-15 14:12:02,327 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1434344795088_0005: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=5
2015-06-15 14:12:03,067 INFO [Socket Reader #1 for port 47209] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1434344795088_0005 (auth:SIMPLE)
2015-06-15 14:12:03,079 INFO [IPC Server handler 2 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1434344795088_0005_r_000003 asked for a task
2015-06-15 14:12:03,079 INFO [IPC Server handler 2 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1434344795088_0005_r_000003 given task: attempt_1434344795088_0005_r_000000_0
2015-06-15 14:12:04,185 INFO [IPC Server handler 1 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request from attempt_1434344795088_0005_r_000000_0. startIndex 0 maxEvents 10000
2015-06-15 14:12:04,363 INFO [IPC Server handler 2 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1434344795088_0005_r_000000_0 is : 0.0
2015-06-15 14:12:04,382 INFO [IPC Server handler 6 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1434344795088_0005_r_000000_0 is : 0.0
2015-06-15 14:12:04,769 INFO [IPC Server handler 7 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1434344795088_0005_r_000000_0 is : 0.0
2015-06-15 14:12:04,772 FATAL [IPC Server handler 9 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1434344795088_0005_r_000000_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:222)
	... 7 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:189)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:220)
	... 7 more
Caused by: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserializeText(BinarySortableSerDe.java:550)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:285)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:185)
	... 8 more

2015-06-15 14:12:04,772 INFO [IPC Server handler 9 on 47209] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1434344795088_0005_r_000000_0: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:222)
	... 7 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:189)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:220)
	... 7 more
Caused by: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserializeText(BinarySortableSerDe.java:550)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:285)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:185)
	... 8 more

2015-06-15 14:12:04,773 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1434344795088_0005_r_000000_0: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from x1x50x48x49x53x45x48x54x45x49x53x32x49x52x58x49x49x58x52x53x0x1x50x48x49x53x45x48x54x45x48x51x0x1x50x48x49x53x45x48x54x45x48x51x32x49x53x58x49x57x58x48x49x46x0x1x50x50x48x46x57x48x46x50x49x54x46x55x51x0x1x104x116x116x112x58x47x47x50x50x48x46x57x48x46x50x0x1x71x69x84x0x1x187x154x203x138x255 with properties {columns=reducesinkkey0,reducesinkkey1,reducesinkkey2,reducesinkkey3,reducesinkkey4,reducesinkkey5,reducesinkkey6, serialization.lib=org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe, serialization.sort.order=+++++++, columns.types=string,string,string,string,string,string,string}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:222)
	... 7 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:189)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:220)
	... 7 more
Caused by: java.io.EOFException
	at org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserializeText(BinarySortableSerDe.java:550)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:285)
	at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:185)
	... 8 more
이 게시물을
목록
번호 제목 글쓴이 날짜 조회 수
661 [Kerberos인증] /var/log/krb5kdc.log파일 기준으로 인증요청(AS), 티켓확인(TGS)이 진행되는 로그 기록 gooper 2022.09.21 36
660 spark에서 hive table을 읽어 출력하는 예제 소스 총관리자 2017.03.09 37
659 [KUDU] kudu tablet server여러가지 원인에 의해서 corrupted상태가 된 경우 복구방법 gooper 2023.03.28 37
658 bash는 PS1 변수를 통해 프롬프트의 모양을 바꿀 수 있다. 총관리자 2016.03.30 38
657 eclipse 3.1 단축키 정리파일 총관리자 2017.01.02 38
656 [u-Auctions]목록이 1개만 나오는 문제 총관리자 2017.05.29 38
655 [Hue]Hue의 메타정보를 담고 있는 desktop_document테이블과 desktop_document2의 관계 총관리자 2022.05.09 38
654 [TLS/SSL]Cloudera 6.3.4기준 Oozie Web UI TLS설정 항목및 설정값 총관리자 2022.05.13 38
653 github에 있는 프로젝트와 로컬에서 작업한 프로젝트 합치기 총관리자 2016.11.22 40
652 DeviceType이 o:motion-sensor_33 이거나 o:motion-sensor_32 경우의 sparql문장은 다음과 같다. 총관리자 2017.08.16 40
651 커리 변경 이벤트를 처리하기 위한 구현클래스 총관리자 2016.07.21 41
650 [oracle]10자리 timestamp값을 날짜로 변환하는 방법 총관리자 2022.04.14 41
649 magento2 설치후 초기화면이 깨지는 문제 file 총관리자 2017.01.31 42
648 windows 혹은 mac에서 docker설치하기 위한 파일 총관리자 2017.10.13 42
647 RDF4J의 rdf4j-server.war가 제공하는 RESTFul API를 이용한 CRUD테스트(트랜잭션처리) 총관리자 2017.08.30 43
646 python실행시 ValueError: zero length field name in format오류 해결방법 총관리자 2016.05.27 44
645 Spark Streaming 코드레벨단에서의 성능개선 총관리자 2016.10.31 44
644 원격의 origin/master를 기준으로 dev branch를 만들어 작업후 원격의 origin/dev에 push하는 방법 file 총관리자 2016.11.22 44
643 new Gson().toJson(new ObjectId())을 사용하면 값이 다르게 나오는 경우가 있음 총관리자 2016.12.23 44
642 특정 단계의 commit상태로 만들기(이렇게 하면 중간에 반영된 모든 commit를 history가 삭제된다) 총관리자 2016.11.17 45
쓰기 태그

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로