事故现场：Error: GC overhead limit exceeded
第一反应：数据量是否猛增 ？ select dt,count(*) from ods.ods_x where dt>='2018-03-01' group by dt order by dt desc; 然而并没有
第二反应：split字段分布不均 ？ 因为sqoop 抽取的原理是，先求min(split 字段)，max(split 字段), 然后平均分，每个map抽相应范围的数据量。如果某一天
split字段有个跳跃式的增长，比如一个量级，虽然数据量没有变化，但是导致split分布不均，有些map分到的数据量过大，有些却几乎没有。
现场还原1：
18/03/14 02:05:32 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "x" 18/03/14 02:05:32 INFO db.IntegerSplitter: split 条件 1=1 and 2=513601701 18/03/13 02:05:34 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN("id"), MAX("id") FROM "x" 18/03/13 02:05:34 INFO db.IntegerSplitter: split 条件 1=1 and 2=10219211
现场还原2:
hive (temp)> dfs -du -h /data/external_table/ods/ods_x/dt=2018-03-12; 0 0 /data/external_table/ods/ods_x/dt=2018-03-12/_SUCCESS 38.0 M 113.9 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00000.snappy 30.9 M 92.7 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00001.snappy 47.2 M 141.7 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00002.snappy 45.6 M 136.9 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00003.snappy 43.8 M 131.4 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00004.snappy 40.0 M 120.0 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00005.snappy 41.5 M 124.6 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00006.snappy 33.6 M 100.9 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00007.snappy 38.5 M 115.6 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00008.snappy 41.0 M 123.1 M /data/external_table/ods/ods_x/dt=2018-03-12/part-m-00009.snappy hive (temp)> dfs -du -h /data/external_table/ods/ods_x/dt=2018-03-13; 0 0 /data/external_table/ods/ods_x/dt=2018-03-13/_SUCCESS 93.3 M 280.0 M /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00000.snappy 113.3 M 340.0 M /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00001.snappy 100.4 M 301.3 M /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00002.snappy 95.4 M 286.1 M /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00003.snappy 106.6 K 319.8 K /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00004.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00005.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00006.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00007.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00008.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00009.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00010.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00011.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00012.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00013.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00014.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00015.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00016.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00017.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00018.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00019.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00020.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00021.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00022.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00023.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00024.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00025.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00026.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00027.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00028.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00029.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00030.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00031.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00032.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00033.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00034.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00035.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00036.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00037.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00038.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00039.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00040.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00041.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00042.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00043.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00044.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00045.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00046.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00047.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00048.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00049.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00050.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00051.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00052.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00053.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00054.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00055.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00056.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00057.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00058.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00059.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00060.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00061.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00062.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00063.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00064.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00065.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00066.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00067.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00068.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00069.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00070.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00071.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00072.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00073.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00074.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00075.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00076.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00077.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00078.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00079.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00080.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00081.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00082.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00083.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00084.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00085.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00086.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00087.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00088.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00089.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00090.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00091.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00092.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00093.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00094.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00095.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00096.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00097.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00098.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00099.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00100.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00101.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00102.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00103.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00104.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00105.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00106.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00107.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00108.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00109.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00110.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00111.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00112.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00113.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00114.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00115.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00116.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00117.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00118.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00119.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00120.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00121.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00122.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00123.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00124.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00125.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00126.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00127.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00128.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00129.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00130.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00131.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00132.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00133.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00134.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00135.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00136.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00137.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00138.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00139.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00140.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00141.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00142.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00143.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00144.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00145.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00146.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00147.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00148.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00149.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00150.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00151.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00152.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00153.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00154.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00155.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00156.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00157.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00158.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00159.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00160.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00161.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00162.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00163.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00164.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00165.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00166.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00167.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00168.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00169.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00170.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00171.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00172.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00173.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00174.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00175.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00176.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00177.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00178.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00179.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00180.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00181.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00182.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00183.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00184.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00185.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00186.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00187.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00188.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00189.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00190.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00191.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00192.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00193.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00194.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00195.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00196.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00197.snappy 4 12 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00198.snappy 260 780 /data/external_table/ods/ods_x/dt=2018-03-13/part-m-00199.snappy
解决方案：
1，替换split字段(该字段要有索引，不然抽取异常慢)
2，split字段应该选择没有业务含义的自增字段
3，与业务开发沟通，寻求解决方案。
posted on
浙公网安备 33010602011771号