Sqoop将MySQL的表数据同步到HDFS（五）基于配置文件执行

新建脚本文件

[root@centos02 opt]# touch TAdminLog_parquet 
[root@centos02 opt]# vim TAdminLog_parquet

import 
--connect 
jdbc:mysql://centos02:3306/OfficialCashMid 
--driver 
com.mysql.cj.jdbc.Driver 
--username 
root 
--password 
sa123_ADMIN. 
--table 
tadminoperationlog 
--m 
2 
--target-dir 
/jdbcHDFS/TAdminLog_parquet 
--as-parquetfile

 

执行文件

[root@centos02 opt]# sqoop --options-file /opt/TAdminLog_parquet 
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/bigdata/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/09/05 23:22:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/bigdata/hadoop/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/bigdata/hive/hive-2.3.4/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/09/05 23:22:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/09/05 23:22:41 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
19/09/05 23:22:41 INFO manager.SqlManager: Using default fetchSize of 1000
19/09/05 23:22:41 INFO tool.CodeGenTool: Beginning code generation
19/09/05 23:22:41 INFO tool.CodeGenTool: Will generate java class as codegen_tadminoperationlog
19/09/05 23:22:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/bigdata/hadoop/hadoop-2.8.5
注: /tmp/sqoop-root/compile/a265e3dfbf8750e4b0c8f71dae4d5310/codegen_tadminoperationlog.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
19/09/05 23:22:54 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/a265e3dfbf8750e4b0c8f71dae4d5310/codegen_tadminoperationlog.jar
19/09/05 23:22:54 INFO mapreduce.ImportJobBase: Beginning import of tadminoperationlog
19/09/05 23:22:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/09/05 23:22:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM tadminoperationlog AS t WHERE 1=0
19/09/05 23:22:56 INFO conf.HiveConf: Found configuration file file:/opt/bigdata/hive/hive-2.3.4/conf/hive-site.xml
19/09/05 23:22:58 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/09/05 23:22:58 INFO client.RMProxy: Connecting to ResourceManager at centos02/192.168.122.1:8032
19/09/05 23:23:06 INFO db.DBInputFormat: Using read commited transaction isolation
19/09/05 23:23:06 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(FID), MAX(FID) FROM tadminoperationlog
19/09/05 23:23:06 INFO db.IntegerSplitter: Split size: 16058; Num splits: 2 from: 21 to: 32138
19/09/05 23:23:06 INFO mapreduce.JobSubmitter: number of splits:2
19/09/05 23:23:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1567635616063_0003
19/09/05 23:23:07 INFO impl.YarnClientImpl: Submitted application application_1567635616063_0003
19/09/05 23:23:07 INFO mapreduce.Job: The url to track the job: http://centos02:8088/proxy/application_1567635616063_0003/
19/09/05 23:23:07 INFO mapreduce.Job: Running job: job_1567635616063_0003
19/09/05 23:23:21 INFO mapreduce.Job: Job job_1567635616063_0003 running in uber mode : false
19/09/05 23:23:21 INFO mapreduce.Job:  map 0% reduce 0%
19/09/05 23:24:13 INFO mapreduce.Job:  map 100% reduce 0%
19/09/05 23:24:17 INFO mapreduce.Job: Job job_1567635616063_0003 completed successfully
19/09/05 23:24:17 INFO mapreduce.Job: Counters: 30
    File System Counters
        FILE: Number of bytes read=0
        FILE: Number of bytes written=361416
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=22036
        HDFS: Number of bytes written=324626
        HDFS: Number of read operations=100
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=20
    Job Counters 
        Launched map tasks=2
        Other local map tasks=2
        Total time spent by all maps in occupied slots (ms)=96361
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=96361
        Total vcore-milliseconds taken by all map tasks=96361
        Total megabyte-milliseconds taken by all map tasks=98673664
    Map-Reduce Framework
        Map input records=12122
        Map output records=12122
        Input split bytes=206
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=713
        CPU time spent (ms)=21130
        Physical memory (bytes) snapshot=443764736
        Virtual memory (bytes) snapshot=4283305984
        Total committed heap usage (bytes)=214433792
    File Input Format Counters 
        Bytes Read=0
    File Output Format Counters 
        Bytes Written=0
19/09/05 23:24:17 INFO mapreduce.ImportJobBase: Transferred 317.0176 KB in 79.4203 seconds (3.9916 KB/sec)
19/09/05 23:24:17 INFO mapreduce.ImportJobBase: Retrieved 12122 records.
[root@centos02 opt]# 

 

查看同步过来的数据文件 

 

[root@centos02 opt]# hdfs dfs -ls /jdbcHDFS/TAdminLog_parquet
Found 4 items
drwxr-xr-x   - root supergroup          0 2019-09-05 23:22 /jdbcHDFS/TAdminLog_parquet/.metadata
drwxr-xr-x   - root supergroup          0 2019-09-05 23:24 /jdbcHDFS/TAdminLog_parquet/.signals
-rw-r--r--   1 root supergroup     188592 2019-09-05 23:24 /jdbcHDFS/TAdminLog_parquet/0d89c8ee-97bb-488d-a9e2-ab65f637c20a.parquet
-rw-r--r--   1 root supergroup     130850 2019-09-05 23:24 /jdbcHDFS/TAdminLog_parquet/6c5a137e-917b-4eb2-b1c3-da0e4322a38d.parquet
[root@centos02 opt]# 
[root@centos02 sbin]# hdfs dfs -cat /jdbcHDFS/TAdminLog_parquet/0d89c8ee-97bb-488d-a9e2-ab65f637c20a.parquet
[root@centos02 sbin]# hdfs dfs -cat /jdbcHDFS/TAdminLog_parquet/6c5a137e-917b-4eb2-b1c3-da0e4322a38d.parquet