Sqoop 的安装使用

1:下载地址 http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/

2:上传压缩包

　　scp sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz root@hadoop2:/opt/software

3:解压并重命名

　　tar -zxvf  sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt/module

　　mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha/ sqoop

4:修改配置文件

　　进入目录 sqoop/conf

　　mv sqoop-env-template.sh sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/module/hadoop-2.10.1

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/module/hadoop-2.10.1

#set the path to where bin/hbase is available
#export HBASE_HOME=/opt/module/hbase-2.3.3

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/module/hive1.2.1


#Set the path for where zookeper config dir is
export ZOOKEEPER_HOME=/opt/module/zookeeper3.5.8
export ZOOCFGDIR=/opt/module/zookeeper3.5.8/conf

6:验证sqoop是否配置正确

　　进入bin目录下执行 ./sqoop help

　　出现如下页面则配置成功 出现的警告是因为没有配置hbase、hcatlog等

Warning: /opt/module/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/module/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/19 11:25:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

7:测试sqoop是否能够连上mysql

　　./sqoop list-databases --connect jdbc:mysql://hadoop2:3306 --username root --password root

　　能够看到databases说明连接成功

Warning: /opt/module/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/module/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
21/04/19 12:02:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
21/04/19 12:02:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/04/19 12:02:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
mysql
information_schema
performance_schema
sys
gmall

到此 sqoop的部署完成

8:导入测试

　　target-dir 上传hadoop位置　　

　　num-mappers 启动mapper的个数，不设置时默认为4个　　

　　fields-terminated-by 分隔符

　　split-by 一那个字段分区，如果mapper为1是否设置效果都相同

./sqoop import \
    --connect jdbc:mysql://hadoop2:3306/gmall \ 
    --username root \
    --password root \
    --table user_info \ 
    --columns id,login_name \
    --where 'id >= 10 and id <= 30' \
    --target-dir /test \
    --delete-target-dir \
    --num-mappers 1 \
    --fields-terminated-by '\t' \
    --split-by id

　　--query 后面可以直接写sql