Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.

hbase Master rejected startup because clock is out of sync 오류 해결방법

총관리자 2016.05.03 11:01 조회 수 : 24223

HMaster서버와 HRegionServer간에 시간차이가 30000ms 이상나면 HRegionServer가 기동하지 못하하게 되는데

이때는 서버간의 시간을 맞춰줄 필요가 있다.

rdate -s time.bora.net를 각 서버에서 실행해주면 time.bora.net에서 제공하는 시간으로 동기화 된다.

혹은

ntpdate time.bora.net를 실행한다.

---------------------------------------------------------------------오류내용---------------------------------------------

2016-05-03 10:47:37,007 FATAL [regionserver/gsda3/xxx.xxx.xxx.41:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master. Time difference of 67321ms > max allowed of 30000ms

at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388)

at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262)

at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:322)

at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

at java.lang.Thread.run(Thread.java:745)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325)

at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2271)

at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server gsda3,16020,1462240056111 has been rejected; Reported time is too far out of sync with master. Time difference of 67321ms > max allowed of 30000ms