Hbase部署报错问题记录

各节点节点时间不一致

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoopslave2,60020,1372320861420has been rejected; Reported time is too far out of sync with master.  Time difference of 143732ms > max allowed of 30000ms         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)         at java.lang.reflect.Constructor.newInstance(Constructor.java:525)         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)         at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2093)         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)         at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoopslave2,60020,1372320861420has been rejected; Reported time is too far out of sync with master.  Time difference of 143732ms > max allowed of 30000ms

在各节点的hbase-site.xml文件中加入下列代码

   <property>
     <name>hbase.master.maxclockskew</name>
     <value>200000</value>
   </property>

Directory is not empty

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/hbase/WALs/slave1,16000,1446046595488-splitting is non empty': Directory is not empty     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)     at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)     at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)     at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)     at java.security.AccessController.doPrivileged(Native Method)     at javax.security.auth.Subject.doAs(Subject.java:415)     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)     at org.apache.hadoop.ipc.Client.call(Client.java:1411)     at org.apache.hadoop.ipc.Client.call(Client.java:1364)     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)     at com.sun.proxy.$Proxy15.delete(Unknown Source)     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)     at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)     at com.sun.proxy.$Proxy16.delete(Unknown Source)     at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)     at com.sun.proxy.$Proxy17.delete(Unknown Source)     at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)     at com.sun.proxy.$Proxy17.delete(Unknown Source)     at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:1726)     at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:588)     at org.apache.hadoop.hdfs.DistributedFileSystem$11.doCall(DistributedFileSystem.java:584)     at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)     at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:584)     at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:297)     at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:400)     at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:373)     at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:295)     at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitLogs(ServerCrashProcedure.java:388)     at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:228)     at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:72)     at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)     at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)     at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1050)     at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:841)     at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:794)     at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)     at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:479)

参考https://issues.apache.org/jira/browse/HBASE-14729，进入hadoop文件系统，删除掉报错的目录或真个WALs

TableExistsException: hbase:namespace

zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1at address=slave1,16020,1428456823337, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1is not online on worker05,16020,1428461295266         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.Java:2740)         at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:859)         at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1137)         at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20862)         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)         at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)         at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)         at java.lang.Thread.run(Thread.java:745)

HMaster启动之后自动挂掉（或非正常重启），并且master的log里出现“TableExistsException: hbase:namespace”字样;
很可能是更换了Hbase的版本过后zookeeper还保留着上一次的Hbase设置，所以造成了冲突.
删除zookeeper信息，重启之后就没问题了

# sh zkCli.sh -server slave1:2181 [zk: slave1:2181(CONNECTED) 0] ls / [zk: slave1:2181(CONNECTED) 0] rmr /hbase [zk: slave1:2181(CONNECTED) 0] quit