bin/start-hbase.sh을 실행하면 아래와 같은 오류가 발생하면서 HMaster 데몬이 기동하지 않는 경우있어,

제시되는 "hbase hbck -fixVersionFile"를 실행하면 실행로그처럼 루프에 빠지게 되는 경우가 있다.

이는 데이타 유실이 발생되어 복구할 수 없는 상태를 나타내므로 zookeeper와 HDFS의 hbase관련 정보및 데이타를 모두 삭제하여야 한다.

(./hbase hbck -fixMeta -fixAssignments를 실행할때도 loop에 빠지게 되면 zookeeper의 /hbase노드를 삭제하고 HDFS상의 /hbase를 삭제한후 bin/start-hbase.sh를 실행한다.)

----hbase hbck -fixVersionFile실행로그---

2016-08-01 14:55:02,619 INFO  [Group Metadata Manager on Broker 1]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
2016-08-01 14:55:16,781 INFO  [main] client.RpcRetryingCaller: Call exception, tries=10, retries=35, started=48182 ms ago, cancelled=false, msg=
2016-08-01 14:55:36,832 INFO  [main] client.RpcRetryingCaller: Call exception, tries=11, retries=35, started=68233 ms ago, cancelled=false, msg=
2016-08-01 14:55:56,942 INFO  [main] client.RpcRetryingCaller: Call exception, tries=12, retries=35, started=88343 ms ago, cancelled=false, msg=
2016-08-01 14:56:17,080 INFO  [main] client.RpcRetryingCaller: Call exception, tries=13, retries=35, started=108481 ms ago, cancelled=false, msg=
2016-08-01 14:56:37,203 INFO  [main] client.RpcRetryingCaller: Call exception, tries=14, retries=35, started=128604 ms ago, cancelled=false, msg=
2016-08-01 14:56:57,328 INFO  [main] client.RpcRetryingCaller: Call exception, tries=15, retries=35, started=148729 ms ago, cancelled=false, msg=
2016-08-01 14:57:17,510 INFO  [main] client.RpcRetryingCaller: Call exception, tries=16, retries=35, started=168911 ms ago, cancelled=false, msg=
2016-08-01 14:57:37,631 INFO  [main] client.RpcRetryingCaller: Call exception, tries=17, retries=35, started=189032 ms ago, cancelled=false, msg=
2016-08-01 14:57:57,818 INFO  [main] client.RpcRetryingCaller: Call exception, tries=18, retries=35, started=209219 ms ago, cancelled=false, msg=
2016-08-01 14:58:17,979 INFO  [main] client.RpcRetryingCaller: Call exception, tries=19, retries=35, started=229380 ms ago, cancelled=false, msg=
2016-08-01 14:58:38,165 INFO  [main] client.RpcRetryingCaller: Call exception, tries=20, retries=35, started=249566 ms ago, cancelled=false, msg=
2016-08-01 14:58:58,282 INFO  [main] client.RpcRetryingCaller: Call exception, tries=21, retries=35, started=269683 ms ago, cancelled=false, msg=
2016-08-01 14:59:18,410 INFO  [main] client.RpcRetryingCaller: Call exception, tries=22, retries=35, started=289811 ms ago, cancelled=false, msg=
2016-08-01 14:59:38,572 INFO  [main] client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=309973 ms ago, cancelled=false, msg=
2016-08-01 14:59:58,699 INFO  [main] client.RpcRetryingCaller: Call exception, tries=24, retries=35, started=330100 ms ago, cancelled=false, msg=
2016-08-01 15:00:18,800 INFO  [main] client.RpcRetryingCaller: Call exception, tries=25, retries=35, started=350201 ms ago, cancelled=false, msg=
2016-08-01 15:00:38,807 INFO  [main] client.RpcRetryingCaller: Call exception, tries=26, retries=35, started=370208 ms ago, cancelled=false, msg=
2016-08-01 15:00:58,982 INFO  [main] client.RpcRetryingCaller: Call exception, tries=27, retries=35, started=390383 ms ago, cancelled=false, msg=
2016-08-01 15:01:19,100 INFO  [main] client.RpcRetryingCaller: Call exception, tries=28, retries=35, started=410501 ms ago, cancelled=false, msg=
2016-08-01 15:01:39,197 INFO  [main] client.RpcRetryingCaller: Call exception, tries=29, retries=35, started=430598 ms ago, cancelled=false, msg=
^C2016-08-01 15:01:54,907 INFO  [Thread-6] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x356353b4f980084
2016-08-01 15:01:54,910 INFO  [Thread-6] zookeeper.ZooKeeper: Session: 0x356353b4f980084 closed
2016-08-01 15:01:54,910 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-08-01 15:01:55,221 INFO  [Thread-6] util.HBaseFsck: Finishing hbck

------------------bin/start-hbase.sh실행시 오류내용------------

2016-08-01 14:47:24,312 INFO  [main] master.HMaster: Adding backup master ZNode /hbase/backup-masters/sda1,16000,1470030443501
2016-08-01 14:47:24,354 INFO  [sda1:16000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/sda1,16000,1470030443501 from backup master directory
2016-08-01 14:47:24,356 INFO  [sda1:16000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=sda1,16000,1470030443501
2016-08-01 14:47:24,378 INFO  [master/sda1/XXX.XXX.XXX.43:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xea78210 connecting to ZooKeeper ensemble=sda1:2181,sda2:2181,sda3:2181
2016-08-01 14:47:24,379 INFO  [master/sda1/XXX.XXX.XXX.43:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=sda1:2181,sda2:2181,sda3:2181 sessionTimeout=90000 watcher=hconnection-0xea
782100x0, quorum=sda1:2181,sda2:2181,sda3:2181, baseZNode=/hbase
2016-08-01 14:47:24,379 INFO  [master/sda1/XXX.XXX.XXX.43:16000-SendThread(sda3:2181)] zookeeper.ClientCnxn: Opening socket connection to server sda3/XXX.XXX.XXX.31:2181. Will not attempt to authenticate u
sing SASL (unknown error)
2016-08-01 14:47:24,379 INFO  [master/sda1/XXX.XXX.XXX.43:16000-SendThread(sda3:2181)] zookeeper.ClientCnxn: Socket connection established to sda3/XXX.XXX.XXX.31:2181, initiating session
2016-08-01 14:47:24,382 INFO  [master/sda1/XXX.XXX.XXX.43:16000-SendThread(sda3:2181)] zookeeper.ClientCnxn: Session establishment complete on server sda3/XXX.XXX.XXX.31:2181, sessionid = 0x356353b4f98007f
, negotiated timeout = 40000
2016-08-01 14:47:24,395 INFO  [master/sda1/XXX.XXX.XXX.43:16000] regionserver.HRegionServer: ClusterId : 2be4df46-db8b-4fc7-a529-12e571444d54
2016-08-01 14:47:24,436 FATAL [sda1:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further informatio
n about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:677)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:650)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:183)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1652)
        at java.lang.Thread.run(Thread.java:745)
2016-08-01 14:47:24,438 FATAL [sda1:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further informatio
n about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:677)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:650)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:183)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1652)
        at java.lang.Thread.run(Thread.java:745)
2016-08-01 14:47:24,438 INFO  [sda1:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2016-08-01 14:47:24,438 INFO  [master/sda1/XXX.XXX.XXX.43:16000] regionserver.HRegionServer: Stopping infoServer
2016-08-01 14:47:24,439 INFO  [master/sda1/XXX.XXX.XXX.43:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2016-08-01 14:47:24,540 INFO  [master/sda1/XXX.XXX.XXX.43:16000] regionserver.HRegionServer: stopping server sda1,16000,1470030443501
2016-08-01 14:47:24,540 INFO  [master/sda1/XXX.XXX.XXX.43:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x356353b4f98007f
2016-08-01 14:47:24,542 INFO  [master/sda1/XXX.XXX.XXX.43:16000] zookeeper.ZooKeeper: Session: 0x356353b4f98007f closed
2016-08-01 14:47:24,542 INFO  [master/sda1/XXX.XXX.XXX.43:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-08-01 14:47:24,542 INFO  [master/sda1/XXX.XXX.XXX.43:16000] regionserver.HRegionServer: stopping server sda1,16000,1470030443501; all regions closed.
2016-08-01 14:47:24,542 INFO  [master/sda1/XXX.XXX.XXX.43:16000] hbase.ChoreService: Chore service for: sda1,16000,1470030443501 had [] on shutdown
2016-08-01 14:47:24,545 INFO  [master/sda1/XXX.XXX.XXX.43:16000] ipc.RpcServer: Stopping server on 16000
2016-08-01 14:47:24,545 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2016-08-01 14:47:24,545 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-08-01 14:47:24,545 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping