hadoop系列 第二坑： hive hbase关联表问题

关键词：

hive创建表卡住了

创建hive和hbase关联表卡住了

 

其实针对这一问题在info级别的日志下是看出哪里有问题的（为什么只能在debug下才能看见呢，不太理解开发者的想法）。

以调试模式进入hive客户端：

hive --hiveconf hive.root.logger=DEBUG,console

执行创建关联表的语句

create table IF NOT EXISTS event_content_temp
(id string,pack string,activity string,title string, sx string, dx string, num bigint) 
stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' 
with serdeproperties ("hbase.columns.mapping" = ":key,cf:pack,cf:activity,cf:title,cf:sx,cf:dx,cf:num") 
tblproperties ("hbase.table.name" = "event_content_temp");

显示错误日志：

15/06/30 09:20:00 INFO util.RetryCounter: Sleeping 4000ms before retry #2...
15/06/30 09:20:00 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
15/06/30 09:20:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)
15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:663)
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:430)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104)
15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:674)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:438)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104)
Interrupting... Be patient, this might take some time.
15/06/30 09:20:01 INFO CliDriver: Interrupting... Be patient, this might take some time.

很显然是zookeeper的问题，可是在ClouderaManager中已经设置了呀，核对之后发现只有Hive Service Configuration Safety Valve for hive-site.xml里加了属性

<property>
  <name>hive.aux.jars.path</name>  
  <value>
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar,
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hbase.jar,
         file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar
  </value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-xxx.cn-north-1.compute.internal:2181,ip-xxxx.cn-north-1.compute.internal:2181,ip-xxxx-112.cn-north-1.compute.internal:2181,ip-xxxx-108.cn-north-1.compute.internal:2181,ip-xxxx111.cn-north-1.compute.internal:2181</value>
</property>

Hive Client Configuration Safety Valve for hive-site.xml里没有添加，这里很好理解，hive服务里配置了hbase的zk，但是hive客户端没有配置（hive-site.xml），建议二者配置一致！

配置完之后，对于关联表的操作都可以顺利完成。

 

总结：建议hive服务端和客户端的hive-site.xml一样

另外，我建了个QQ群：305994766，希望对大数据、算法研发、系统架构感兴趣的朋友能够加入进来，大家一起学习，共同进步（进群请说明自己的公司-职业-昵称）

 

posted on 2015-07-02 00:10  leocook  阅读(4513)  评论(0)  编辑  收藏  举报

导航