MySQL数据库表 导入到 HDFS指定目录

 Sqoop 执行脚本

[root@centos02 centos02]# cd $SQOOP_HOME
[root@centos02 sqoop-1.4.7]# cd ./bin
[root@centos02 bin]# 

[root@centos02 bin]# sqoop import --connect jdbc:mysql://centos02:3306/OfficialCashMid --username root --password sa123_***** --table tadminoperationlog --m 1 --target-dir /dirTadminoperationlog 
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/09/03 10:59:39 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
19/09/03 10:59:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/09/03 10:59:40 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/09/03 10:59:40 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
19/09/03 10:59:40 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tadminoperationlog` AS t LIMIT 1
19/09/03 10:59:41 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tadminoperationlog` AS t LIMIT 1
19/09/03 10:59:41 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/bigdata/hadoop/hadoop-2.8.5
注: /tmp/sqoop-root/compile/f2781e5449075de95fc5fe2f7fc872d8/tadminoperationlog.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
19/09/03 10:59:47 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f2781e5449075de95fc5fe2f7fc872d8/tadminoperationlog.jar
19/09/03 10:59:47 WARN manager.MySQLManager: It looks like you are importing from mysql.
19/09/03 10:59:47 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
19/09/03 10:59:47 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
19/09/03 10:59:47 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
19/09/03 10:59:47 INFO mapreduce.ImportJobBase: Beginning import of tadminoperationlog
19/09/03 10:59:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/09/03 10:59:48 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/09/03 10:59:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/09/03 10:59:49 INFO client.RMProxy: Connecting to ResourceManager at centos02/192.168.122.1:8032
19/09/03 10:59:58 INFO db.DBInputFormat: Using read commited transaction isolation
19/09/03 10:59:58 INFO mapreduce.JobSubmitter: number of splits:1
19/09/03 10:59:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1567472733544_0001
19/09/03 11:00:00 INFO impl.YarnClientImpl: Submitted application application_1567472733544_0001
19/09/03 11:00:01 INFO mapreduce.Job: The url to track the job: http://centos02:8088/proxy/application_1567472733544_0001/
19/09/03 11:00:01 INFO mapreduce.Job: Running job: job_1567472733544_0001
19/09/03 11:00:41 INFO mapreduce.Job: Job job_1567472733544_0001 running in uber mode : false
19/09/03 11:00:41 INFO mapreduce.Job:  map 0% reduce 0%
19/09/03 11:01:36 INFO mapreduce.Job:  map 100% reduce 0%
19/09/03 11:01:38 INFO mapreduce.Job: Job job_1567472733544_0001 completed successfully
19/09/03 11:01:38 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=178920
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=87
        HDFS: Number of bytes written=3753700
        HDFS: Number of read operations=4
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Other local map tasks=1
        Total time spent by all maps in occupied slots (ms)=49905
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=49905
        Total vcore-milliseconds taken by all map tasks=49905
        Total megabyte-milliseconds taken by all map tasks=51102720
    Map-Reduce Framework
        Map input records=12122
        Map output records=12122
        Input split bytes=87
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=248
        CPU time spent (ms)=7740
        Physical memory (bytes) snapshot=189337600
        Virtual memory (bytes) snapshot=2121957376
        Total committed heap usage (bytes)=92274688
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=3753700
19/09/03 11:01:38 INFO mapreduce.ImportJobBase: Transferred 3.5798 MB in 108.9399 seconds (33.649 KB/sec)
19/09/03 11:01:38 INFO mapreduce.ImportJobBase: Retrieved 12122 records.
[root@centos02 bin]# 

 

查看目录

[root@centos02 hadoop-2.8.5]# hdfs dfs -ls /dirTadminoperationlog
Found 2 items
-rw-r--r--   1 root supergroup          0 2019-09-03 11:01  /dirTadminoperationlog/_SUCCESS
-rw-r--r--   1 root supergroup    3753700 2019-09-03 11:01  /dirTadminoperationlog/part-m-00000
[root@centos02 hadoop-2.8.5]# 

导入完成的表tadminoperationlog表导入一个以逗号(,)分隔的文件，即SCV逗号分割值。
默认情况下,带有表名的目录会被创建在HDFS中,它将存储tadminoperationlog数据的CSV文件。