메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


Mariadb 10.0.1에 Hive meta정보를 저장하기 위한 db(hive_metastore_db)를 생성하고 schematool -initSchema -dbType mysql --verbose 를 실행하면 정상적으로 진행되는듯 하다가 마지막 부분에 아래와 같은 오류(오류내용#1)가 발생하지만 db내용을 보면 필요한 테이블이 모두 생성되어 있다.


그래서 hive --service metastore를 실행하면 "Exception in thread "main" MetaException(message:Version information not found in metastore. )"오류(오류내용#2)가 발생하는데 이유는 오류내용#1부분에서 보이는것처럼 hive-schema-2.1.0.mysql.sql의 끝부분에서 Hive version정보를 Version테이블에 입력하는데 이 부분을 수행하지 못해서 발생하는 문제이다.

이때는 해당 sql을 별도로 실행해주면된다.

(버젼정보입력 sql : INSERT INTO VERSION (VER_ID, SCHEMA_VERSION, VERSION_COMMENT) VALUES (1, '2.1.0', 'Hive release version 2.1.0');


-----------오류내용#1(schematool -initSchema -dbType mysql --verbose 실행시 발생하는 오류내용)----------------------

... 중간 생략
No rows affected (0.005 seconds)
0: jdbc:mysql://gsda3:3306/hive_metastore_db> CREATE INDEX `CONSTRAINTS_PARENT_TABLE_ID_INDEX` ON KEY_CONSTRAINTS (`PARENT_TBL_ID`) USING BTREE
No rows affected (0.003 seconds)
0: jdbc:mysql://gsda3:3306/hive_metastore_db> CREATE TABLE TXNS ( TXN_ID bigint PRIMARY KEY, TXN_STATE char(1) NOT NULL, TXN_STARTED bigint NOT NULL, TXN_LAST_HEARTBEAT bigint NOT NULL, TXN_USER varchar(128) NOT NULL, TXN_HOST varchar(128) NOT NULL, TXN_AGENT_INFO varchar(128), TXN_META_INFO varchar(128), TXN_HEARTBEAT_COUNT int ) ENGINE=InnoDB DEFAULT CHARSET=latin1
No rows affected (0.004 seconds)
0: jdbc:mysql://gsda3:3306/hive_metastore_db> CREATE TABLE TXN_COMPONENTS ( TC_TXNID bigint NOT NULL, TC_DATABASE varchar(128) NOT NULL, TC_TABLE varchar(128) NOT NULL, TC_PARTITION varchar(767), TC_OPERATION_TYPE char(1) NOT NULL, FOREIGN KEY (TC_TXNID) REFERENCES TXNS (TXN_ID) ) ENGINE=InnoDB DEFAULT CHARSET=latin1
No rows affected (0.007 seconds)
0: jdbc:mysql://gsda3:3306/hive_metastore_db> CREATE TABLE COMPLETED_TXN_COMPONENTS ( CTC_TXNID bigint NOT NULL, CTC_DATABASE varchar(128) NOT NULL, CTC_TABLE varchar(128), CTC_PARTITION varchar(767) ) ENGINE=InnoDB DEFAULT CHARSET=latin1
No rows affected (0.005 seconds)
0: jdbc:mysql://gsda3:3306/hive_metastore_db> CREATE TABLE NEXT_TXN_ID ( NTXN_NEXT bigint NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1
No rows affected (0.004 seconds)
   jdbc:mysql://gsda3:3306/hive_metastore_db> INSERT INTO NEXT_TXN_ID VALUES(1)
Error: (conn:18) Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.
Query is : INSERT INTO NEXT_TXN_ID VALUES(1) (state=HY000,code=1665)

Closing: 0: jdbc:mysql://gsda3:3306/hive_metastore_db?createDatabaseIfNotExist=true
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!
        at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
        at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
        at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.io.IOException: Schema script failed, errorcode 2
        at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
        at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
        at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
        ... 8 more
*** schemaTool failed ***


--------------------------오류내용#2(nohup hive --service metastore & 실행시 발생하는 오류내용)----------------------

Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/svc/apps/gsda/bin/hadoop/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/svc/apps/gsda/bin/hadoop/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
MetaException(message:Version information not found in metastore. )
        at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
        at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
        at com.sun.proxy.$Proxy18.verifySchema(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6484)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6479)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6737)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6664)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Exception in thread "main" MetaException(message:Version information not found in metastore. )
        at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
        at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
        at com.sun.proxy.$Proxy18.verifySchema(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6484)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6479)

번호 제목 글쓴이 날짜 조회 수
70 [sentry]role부여후 테이블명이 변경되어 오류가 발생할때 조치방법 총관리자 2018.10.16 215
69 hive 0.13.1 설치 + meta정보는 postgresql 9.3에 저장 총관리자 2015.04.30 227
68 AIX 7.1에서 hive실행시 "hive: line 86: readlink: command not found" 오류가 발생시 임시 조치사항 총관리자 2016.09.25 233
67 Scala에서 countByWindow를 이용하기(예제) 총관리자 2018.03.08 235
66 hive metastore db중 TBLS, TABLE_PARAMS테이블 설명 총관리자 2021.10.22 259
65 Windows7 64bit 환경에서 Apache Spark 2.2.0 설치하기 총관리자 2017.07.26 259
64 Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging 발생시 조치사항 총관리자 2017.04.19 284
63 Spark에서 KafkaUtils.createStream()를 이용하여 이용하여 kafka topic에 접근하여 객채로 저장된 값을 가져오고 처리하는 예제 소스 총관리자 2017.04.26 292
62 spark-submit으로 spark application실행하는 다양한 방법 총관리자 2016.05.25 303
61 Apache Spark와 Drools를 이용한 CEP구현 테스트 총관리자 2016.07.15 342
60 Hive MetaStore Server기동시 Could not create "increment"/"table" value-generation container SEQUENCE_TABLE since autoCreate flags do not allow it. 오류발생시 조치사항 총관리자 2017.05.03 345
59 hive metadata(hive, impala, kudu 정보가 있음) 테이블에서 db, table, owner, location를 조회하는 쿼리 총관리자 2020.02.07 380
58 2개 data를 join하고 마지막으로 code정보를 join하여 결과를 얻는 mr 프로그램 총관리자 2014.06.30 408
57 Permission denied: user=hadoop, access=EXECUTE, inode="/tmp":root:supergroup:drwxrwx--- 오류해결방법 총관리자 2015.05.17 412
56 conda를 이용한 jupyterhub(v0.9)및 jupyter설치 (v4.4.0) 총관리자 2018.07.30 421
55 Tracking URL = N/A 가발생하는 경우 - 환경설정값을 잘못설정하는 경우에 발생함 총관리자 2015.06.17 423
54 spark-sql실행시 The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH오류 발생시 조치사항 총관리자 2016.06.09 441
53 Spark 1.6.1 설치후 HA구성 총관리자 2016.05.24 455
52 java.lang.OutOfMemoryError: unable to create new native thread오류 발생지 조치사항 총관리자 2016.10.17 466
» Ubuntu 16.04 LTS에 Hive 2.1.1설치하면서 "Version information not found in metastore"발생하는 오류원인및 조치사항 총관리자 2017.05.03 469

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로