每隔2分钟，就往hive中增量导入数据，自循环实现导入-sqoop

每隔2分钟，就往hive中增量导入数据，自循环实现导入

 1 #!/bin/sh
 2 export SQOOP_HOME=/usr/share/sqoop-1.4.4
 3 hostname="192.168.1.199"
 4 user="root"
 5 password="root"
 6 database="test"
 7 table="tags"
 8 curr_max=0
 9  
10 function db_to_hive(){
11  
12     ${SQOOP_HOME}/bin/sqoop import --connect jdbc:mysql://${hostname}/${database} --username ${user} --password ${password}  --table ${table}   --split-by docid --hive-import --hive-table lan.ding  
13  --fields-terminated-by '\t' --incremental  append  --check-column docid --last-value ${curr_max}   
14     result=`mysql -h${hostname} -u${user} -p${password} ${database}<<EOF 
15 select max(docid) from ${table};
16 EOF`
17 curr_max=`echo $result |awk '{print $2}'`
18 }
19  
20 if [ $# -eq 0 ];then
21     while true
22     do
23       db_to_hive
24       sleep 120
25     done
26     exit
27 fi

 

posted @ 2020-01-20 10:24  Tim&Blog  阅读(1098)  评论(4)  编辑  收藏  举报