使用 sqoop 将 hive 数据导出到 mysql (export)
使用sqoop将hive中的数据传到mysql中
1.新建hive表
hive> create external table sqoop_test(id int,name string,age int) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > STORED AS TEXTFILE > location '/user/hive/external/sqoop_test'; OK Time taken: 0.145 seconds
2.给hive表添加数据
数据如下
1,fz,13
2,test,13
3,dx,18
3.将文件上传到hdfs对应目录下
hadoop fs -put sqoop_test.txt /user/hive/external/sqoop_test/
EFdeMacBook-Pro:testfile FengZhen$ hadoop fs -ls /user/hive/external/sqoop_test/ 17/09/13 10:22:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 1 FengZhen supergroup 26 2017-09-13 10:21 /user/hive/external/sqoop_test/sqoop_test.txt
上传成功
进入hive 命令行可查看到数据
hive> select * from sqoop_test; OK 1 fz 13 2 test 13 3 dx 18 Time taken: 0.089 seconds, Fetched: 3 row(s)
4.在mysql新建表，表结构和hive中的相同
CREATE TABLE `sqoop_test` ( `id` int(11) DEFAULT NULL, `name` varchar(255) DEFAULT NULL, `age` int(11) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1
5.使用sqoop传输数据
sqoop export
--connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test
--export-dir /user/hive/external/sqoop_test --input-fields-terminated-by ,
EFdeMacBook-Pro:bin FengZhen$ sqoop export --connect jdbc:mysql://localhost:3306/sqooptest --username root --password 123qwe --table sqoop_test --export-dir /user/hive/external/sqoop_test --input-fields-terminated-by , Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /Users/FengZhen/Desktop/Hadoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hbase-1.3.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/09/13 10:28:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 17/09/13 10:28:01 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 17/09/13 10:28:01 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 17/09/13 10:28:01 INFO tool.CodeGenTool: Beginning code generation 17/09/13 10:28:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 1 17/09/13 10:28:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `sqoop_test` AS t LIMIT 1 17/09/13 10:28:02 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /Users/FengZhen/Desktop/Hadoop/hadoop-2.8.0 17/09/13 10:28:04 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-FengZhen/compile/7a078053fb0424d718e08c56fc9bab27/sqoop_test.jar 17/09/13 10:28:04 INFO mapreduce.ExportJobBase: Beginning export of sqoop_test 17/09/13 10:28:04 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 17/09/13 10:28:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/13 10:28:04 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 17/09/13 10:28:05 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 17/09/13 10:28:05 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 17/09/13 10:28:05 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 17/09/13 10:28:06 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 17/09/13 10:28:07 INFO input.FileInputFormat: Total input files to process : 1 17/09/13 10:28:07 INFO input.FileInputFormat: Total input files to process : 1 17/09/13 10:28:07 INFO mapreduce.JobSubmitter: number of splits:4 17/09/13 10:28:07 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 17/09/13 10:28:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505268150495_0004 17/09/13 10:28:08 INFO impl.YarnClientImpl: Submitted application application_1505268150495_0004 17/09/13 10:28:08 INFO mapreduce.Job: The url to track the job: http://192.168.1.64:8088/proxy/application_1505268150495_0004/ 17/09/13 10:28:08 INFO mapreduce.Job: Running job: job_1505268150495_0004 17/09/13 10:28:18 INFO mapreduce.Job: Job job_1505268150495_0004 running in uber mode : false 17/09/13 10:28:18 INFO mapreduce.Job: map 0% reduce 0% 17/09/13 10:28:32 INFO mapreduce.Job: map 100% reduce 0% 17/09/13 10:28:32 INFO mapreduce.Job: Job job_1505268150495_0004 completed successfully 17/09/13 10:28:32 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=626576 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=758 HDFS: Number of bytes written=0 HDFS: Number of read operations=19 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job Counters Launched map tasks=4 Data-local map tasks=4 Total time spent by all maps in occupied slots (ms)=45180 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=45180 Total vcore-milliseconds taken by all map tasks=45180 Total megabyte-milliseconds taken by all map tasks=46264320 Map-Reduce Framework Map input records=3 Map output records=3 Input split bytes=671 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=266 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=599785472 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0 17/09/13 10:28:32 INFO mapreduce.ExportJobBase: Transferred 758 bytes in 26.9573 seconds (28.1185 bytes/sec) 17/09/13 10:28:32 INFO mapreduce.ExportJobBase: Exported 3 records.
传输完成，mysql已经有数据了。
