메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


HMaster서버와 HRegionServer간에 시간차이가 30000ms 이상나면 HRegionServer가 기동하지 못하하게 되는데
이때는 서버간의 시간을 맞춰줄 필요가 있다.

rdate -s time.bora.net를 각 서버에서 실행해주면 time.bora.net에서 제공하는 시간으로 동기화 된다.
혹은
ntpdate time.bora.net를 실행한다.



---------------------------------------------------------------------오류내용---------------------------------------------
2016-05-03 10:47:37,007 FATAL [regionserver/gsda3/xxx.xxx.xxx.41:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2271)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1235)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:222)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:323)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2269)
        ... 2 more
2016-05-03 10:47:37,008 FATAL [regionserver/gsda3/xxx.xxx.xxx.41:16020] regionserver.HRegionServer: ABORTING region server gsda3,16020,1462240056111: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2271)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1235)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:222)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:323)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2269)

.................

2016-05-03 10:47:37,022 INFO  [regionserver/gsda3/xxx.xxx.xxx.41:16020] regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda1,16020,1462240056111 has been rejected; Reported time is too far out of sync with master.  Time difference of 67321ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)
        at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)

번호 제목 날짜 조회 수
202 Spark 1.6.1 설치후 HA구성 2016.05.24 1065
201 spark-shell실행시 "A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection."오류가 발생하는 경우 해결방법 2016.05.20 688
» Master rejected startup because clock is out of sync 오류 해결방법 2016.05.03 234
199 kafka broker기동시 brokerId가 달라서 기동에 실패하는 경우 조치방법 2016.05.02 2474
198 kafka 0.9.0.1 for scala 2.1.1 설치및 테스트 2016.05.02 959
197 solr 인스턴스 기동후 shard에 서버가 정상적으로 할당되지 않는 경우 해결책 2016.04.29 1397
196 collection생성시 -shards와 -replicationFactor값을 잘못지정하면 write.lock for client xxx.xxx.xxx.xxx already exists오류가 발생한다. 2016.04.28 231
195 solrdf초기 기동시 "Caused by: java.lang.IllegalAccessError: tried to access field org.apache.solr.handler.RequestHandlerBase.log from class org.gazzax.labs.solrdf.handler.update.RdfUpdateRequestHandler" 오류가 발생시 조치사항 2016.04.22 432
194 solrcloud에 solrdf1.1설치하고 테스트 하기 2016.04.22 686
193 Spark 2.1.1 clustering(5대) 설치(YARN기반) 2016.04.22 2210
192 Hadoop 완벽 가이드 정리된 링크 2016.04.19 831
191 cumulusRDF 1.0.1설치및 "KeyspaceCumulus" keyspace확인하기 file 2016.04.15 6093
190 bin/cassandra -f -R로 startup할때 NullPointerException오류가 나면 조치할 내용 2016.04.14 423
189 Cassandra 3.4(3.10) 설치/설정 (5대로 clustering) 2016.04.11 987
188 딥러닝 수학/알고리즘 '한국어' 강의 2016.04.10 514
187 서버 5대에 solr 5.5.0 설치하고 index data를 HDFS에 저장/search하도록 설치/설정하는 방법(SolrCloud) 2016.04.08 821
186 서버 5대에 solr 5.5.0 설치하고 index data를 HDFS에 저장/search하도록 설치/설정하는 방법 2016.04.08 850
185 failed to read local state, exiting...오류발생시 조치사항 2016.04.06 308
184 elasticsearch 1.3.0에서 rdf및 hadoop plugin설치 2016.04.06 566
183 Elastic Search For Hadoop 2.2.0설치하기(5대 클러스터링) 2016.04.04 566
위로