Hadoop2.7.3环境下Sqoop1.99.5安装

一、安装准备

1.已经装好的hadoop环境是hadoop 2.7.3
参考博客：http://blog.csdn.net/qq_38799155/article/details/77748831

2.下载sqoop的安装包（注意是hadoop200）
下载地址：http://archive.apache.org/dist/sqoop/1.99.5/

二、搭建环境

1.通过xftp，将下载的sqoop-1.99.5-bin-hadoop200.tar.gz上传到hadoop用户的根目录下，然后解压并且移动到sqoop文件夹下;

//解压
tar -zvxf sqoop-1.99.5-bin-hadoop200.tar.gz
//将解压好的sqoop移动到sqoop目录下
mv sqoop-1.99.5-bin-hadoop200 sqoop

2.配置环境变量
2.1改变sqoop/bin下边文件的执行权限

cd sqoop/bin/
chmod a+x *

2.2在hadoop用户下配置：vi .bashrc

添加sqoop环境变量：

# sqoop Environment Variables
export SQOOP_HOME=/home/hadoop/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

保存退出即时生效：source .bashrc

3.修改sqoop配置：

vi /home/hadoop/sqoop/server/conf/sqoop.properties

在文件中找到org.apache.sqoop.submission.engine.mapreduce.configuration.directory，用Hadoop的配置文件目录赋值，最终效果如下：
示图：

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-2.7.3/etc/hadoop

4.修改配置文件：
修改配置文件catalina.properties：

vi /home/hadoop/sqoop/server/conf/catalina.properties

在文件中找到common.loader，在其后添加hadoop相关jar包路径，最终效果如下：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/common/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/common/lib/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/hdfs/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/hdfs/lib/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/httpfs/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/httpfs/bin/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/mapreduce/bin/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/tools/bin/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/*.jar,/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/bin/*.jar

安装中遇到的问题：
我在启动show version –all时，出现GET http://localhost:12000/sqoop/version returned a response status of 404 Not Found
后来查看日志，发现时log4j的原因。
由于在catalina.properties配置文件的common.loader属性中配置引入了所有的hadoop相关jar包，而hadoop的lib目录下是有log4j包的，只是版本和sqoop自带的log4j不一致。
我们需要将/home/hadoop/sqoop/server/webapps/sqoop/WEB-INF/lib目录下的log4j-1.2.16.jar移除即可。

发现还是不能解决，日志仍然抱  Java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory ，于是手动将hadoop的jar包全部copy出来，然后复制到/home/hadoop/sqoop/server/webapps/sqoop/WEB-INF/lib目录下，还原common.loader中修改的内容，然后启动就好了。

5.下载mysql驱动包
下载地址：http://dev.mysql.com/downloads/connector/j/ ，解压后拷贝其中的mysql-connector-java-5.1.38-bin.jar到/home/hadoop/sqoop/server的lib文件夹下。

6.测试
在 hadoop用户下执行此命令：

sqoop2-tool verify

出现如图所示的结果，即为正确

7.sqoop的操作

sqoop2-server start     #启动
sqoop2-server stop      #停止

8.进入客户端交互目录

sqoop2-shell

如图所示，启动成功

具体的一些应用操作，请参考网址：http://sqoop.apache.org/docs/1.99.5/CommandLineClient.html

9.常见的错误

Exception has occurred during processing command 
Exception: org.apache.sqoop.common.SqoopException Message: CLIENT_0001:Server has returned exception
Stack trace:
     at  org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:129)  
     at  org.apache.sqoop.client.request.ResourceRequest (ResourceRequest.java:179)  
     at  org.apache.sqoop.client.request.JobResourceRequest (JobResourceRequest.java:112)  
     at  org.apache.sqoop.client.request.SqoopResourceRequests (SqoopResourceRequests.java:154)  
     at  org.apache.sqoop.client.SqoopClient (SqoopClient.java:481)  
     at  org.apache.sqoop.shell.StartJobFunction (StartJobFunction.java:75)  
     at  org.apache.sqoop.shell.SqoopFunction (SqoopFunction.java:51)  
     at  org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:141)  
     at  org.apache.sqoop.shell.SqoopCommand (SqoopCommand.java:103)  
     at  org.codehaus.groovy.tools.shell.Command$execute (null:-1)  
     at  org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:42)  
     at  org.codehaus.groovy.tools.shell.Command$execute (null:-1)  
     at  org.codehaus.groovy.tools.shell.Shell (Shell.groovy:101)  
     at  org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:-1)  
     at  sun.reflect.GeneratedMethodAccessor24 (null:-1)  
     at  sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)  
     at  java.lang.reflect.Method (Method.java:498)  
     at  org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)  
     at  groovy.lang.MetaMethod (MetaMethod.java:233)  
     at  groovy.lang.MetaClassImpl (MetaClassImpl.java:1054)  
     at  org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:128)  
     at  org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:173)  
     at  sun.reflect.GeneratedMethodAccessor23 (null:-1)  
     at  sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)  
     at  java.lang.reflect.Method (Method.java:498)  
     at  org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:267)  
     at  org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:52)  
     at  org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:141)  
     at  org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:121)  
     at  org.codehaus.groovy.tools.shell.Shell (Shell.groovy:114)  
     at  org.codehaus.groovy.tools.shell.Shell$leftShift$0 (null:-1)  
     at  org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:88)  
     at  org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)  
     at  sun.reflect.GeneratedMethodAccessor21 (null:-1)  
     at  sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)  
     at  java.lang.reflect.Method (Method.java:498)  
     at  org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)  
     at  groovy.lang.MetaMethod (MetaMethod.java:233)  
     at  groovy.lang.MetaClassImpl (MetaClassImpl.java:1054)  
     at  org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:128)  
     at  org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:148)  
     at  org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:100)  
     at  sun.reflect.GeneratedMethodAccessor20 (null:-1)  
     at  sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)  
     at  java.lang.reflect.Method (Method.java:498)  
     at  org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce (PogoMetaMethodSite.java:267)  
     at  org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite (PogoMetaMethodSite.java:52)  
     at  org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:137)  
     at  org.codehaus.groovy.tools.shell.ShellRunner (ShellRunner.groovy:57)  
     at  org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:-1)  
     at  sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:-2)  
     at  sun.reflect.NativeMethodAccessorImpl (NativeMethodAccessorImpl.java:62)  
     at  sun.reflect.DelegatingMethodAccessorImpl (DelegatingMethodAccessorImpl.java:43)  
     at  java.lang.reflect.Method (Method.java:498)  
     at  org.codehaus.groovy.reflection.CachedMethod (CachedMethod.java:90)  
     at  groovy.lang.MetaMethod (MetaMethod.java:233)  
     at  groovy.lang.MetaClassImpl (MetaClassImpl.java:1054)  
     at  org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:128)  
     at  org.codehaus.groovy.runtime.ScriptBytecodeAdapter (ScriptBytecodeAdapter.java:148)  
     at  org.codehaus.groovy.tools.shell.InteractiveShellRunner (InteractiveShellRunner.groovy:66)  
     at  java_lang_Runnable$run (null:-1)  
     at  org.codehaus.groovy.runtime.callsite.CallSiteArray (CallSiteArray.java:42)  
     at  org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:108)  
     at  org.codehaus.groovy.runtime.callsite.AbstractCallSite (AbstractCallSite.java:112)  
     at  org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:463)  
     at  org.codehaus.groovy.tools.shell.Groovysh (Groovysh.groovy:402)  
     at  org.apache.sqoop.shell.SqoopShell (SqoopShell.java:128)  
Caused by: Exception: org.apache.sqoop.common.SqoopException Message: GENERIC_JDBC_CONNECTOR_0005:No column is found to partition data - 
Stack trace:
     at  org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer (GenericJdbcFromInitializer.java:147)  
     at  org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer (GenericJdbcFromInitializer.java:51)  
     at  org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer (GenericJdbcFromInitializer.java:40)  
     at  org.apache.sqoop.driver.JobManager (JobManager.java:448)  
     at  org.apache.sqoop.driver.JobManager (JobManager.java:371)  
     at  org.apache.sqoop.driver.JobManager (JobManager.java:275)  
     at  org.apache.sqoop.handler.JobRequestHandler (JobRequestHandler.java:379)  
     at  org.apache.sqoop.handler.JobRequestHandler (JobRequestHandler.java:115)  
     at  org.apache.sqoop.server.v1.JobServlet (JobServlet.java:96)  
     at  org.apache.sqoop.server.SqoopProtocolServlet (SqoopProtocolServlet.java:79)  
     at  javax.servlet.http.HttpServlet (HttpServlet.java:646)  
     at  javax.servlet.http.HttpServlet (HttpServlet.java:723)  
     at  org.apache.catalina.core.ApplicationFilterChain (ApplicationFilterChain.java:290)  
     at  org.apache.catalina.core.ApplicationFilterChain (ApplicationFilterChain.java:206)  
     at  org.apache.hadoop.security.authentication.server.AuthenticationFilter (AuthenticationFilter.java:594)  
     at  org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter (DelegationTokenAuthenticationFilter.java:291)  
     at  org.apache.hadoop.security.authentication.server.AuthenticationFilter (AuthenticationFilter.java:553)  
     at  org.apache.catalina.core.ApplicationFilterChain (ApplicationFilterChain.java:235)  
     at  org.apache.catalina.core.ApplicationFilterChain (ApplicationFilterChain.java:206)  
     at  org.apache.catalina.core.StandardWrapperValve (StandardWrapperValve.java:233)  
     at  org.apache.catalina.core.StandardContextValve (StandardContextValve.java:191)  
     at  org.apache.catalina.core.StandardHostValve (StandardHostValve.java:127)  
     at  org.apache.catalina.valves.ErrorReportValve (ErrorReportValve.java:103)  
     at  org.apache.catalina.core.StandardEngineValve (StandardEngineValve.java:109)  
     at  org.apache.catalina.connector.CoyoteAdapter (CoyoteAdapter.java:293)  
     at  org.apache.coyote.http11.Http11Processor (Http11Processor.java:861)  
     at  org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler (Http11Protocol.java:606)  
     at  org.apache.tomcat.util.net.JIoEndpoint$Worker (JIoEndpoint.java:489)  
     at  java.lang.Thread (Thread.java:745)  

错误的原因是：数据库里的表没加主键