Error: attempt to setup a Window for datatype VARCHAR 和 Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"b","_col1":"2"},"value":{"_col0":1260}}

环境：

这个问题在我使用CDH搭建的集群上遇到的，主要是建表的时候统一用的VARCHAR类型，不要问为什么，我也不太清楚，总之生产上就是这么做的就是了。然后执行聚合函数的窗口函数的时候报的这个错误。之前也有过这种类似的操作，但是还是第一次碰见这种错误，在我解决这个问题的过程中，出现了两种错误，我都记录了，但是不保证都是因为我说的这个原因。

问题内容：

第一个错误

Vertex failed, vertexName=Reducer 2, vertexId=vertex_1604444511298_0012_18_01, diagnostics=[Task failed, taskId=task_1604444511298_0012_18_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1604444511298_0012_18_01_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"b","_col1":"2"},"value":{"_col0":1260}}
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
    at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
    at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"b","_col1":"2"},"value":{"_col0":1260}}
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:296)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
    ... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"b","_col1":"2"},"value":{"_col0":1260}}
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:364)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:286)
    ... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error: attempt to setup a Window for datatype VARCHAR
    at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:774)
    at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:355)
    ... 17 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error: attempt to setup a Window for datatype VARCHAR
    at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction$WindowingIterator.next(WindowingTableFunction.java:1593)
    at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:373)
    at org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:122)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
    at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1024)
    at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:829)
    at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:703)
    at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:769)
    ... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Internal Error: attempt to setup a Window for datatype VARCHAR
    at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction$SingleValueBoundaryScanner.getScanner(WindowingTableFunction.java:1177)
    at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.getRange(WindowingTableFunction.java:694)
    at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.access$100(WindowingTableFunction.java:70)
    at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction$WindowingIterator.next(WindowingTableFunction.java:1579)
    ... 25 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1604444511298_0012_18_01 [Reducer 2] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
20/11/04 08:48:45 INFO ql.Driver: Completed executing command(queryId=root_20201104084833_72e651e1-ee1d-4767-a102-79e0af819153); Time taken: 11.848 seconds

第二种错误

Internal Error: attempt to setup a Window for datatype VARCHAR

问题分析：

建表信息

CREATE EXTERNAL TABLE IF NOT EXISTS shop (

`db` VARCHAR(100),

`mon` VARCHAR(3),

`day` VARCHAR(3),

`sale` VARCHAR(100)

) COMMENT "商店余额表" PARTITIONED (day string)

ROW FORMAT delimited fields terminated by ','

LOCATION "/data/table/";

数据

INSERT INTO TABLE `shop` PARTITION (day='20201102') VALUES ('a','1','5','200'),('a','2','1','500'),('a','2','3','600'),('a','2','7','400'),('a','3','3','100'),('a','3','6','200'),('a','3','8','400'),('b','1','3','100'),('b','1','7','220'),('b','2','8','500'),('b','2','13','300'),('b','2','27','460'),('b','3','5','100'),('b','3','16','250'),('b','3','18','450');

脚本

select 

    t.dp dp,

    t.m m,

    t.sale_sum amt,

    sum(t.sale_sum) over(partition by t.dp order by t.m) zonge

from (select dp,m,sum(sale) sale_sum from shop group by dp,m) t;

说实话，没什么分析的过程，因为找到日志也是这样的错误信息，所以刚开始就各种网上搜索，但是后来说实话都没解决。最后的最后是我猜想，既然报错VARCHAR还有列的一下信息，会不会是因为的的数据类型呢，抱着试一试的态度搞了一下，还真的成了，刚开始我还是重新建表之后再写的，后来发现直接使用cast就可以了。

解决办法

第一种

修改建表语句

CREATE EXTERNAL TABLE IF NOT EXISTS shop (

`db` VARCHAR(100),

`mon` INT

。。。。。。

第二种

修改脚本

select

t.dp dp,

t.m m,

t.sale_sum amt,

sum(t.sale_sum) over(partition by t.dp order cast(by t.m as int)) zonge

。。。。。

总结

总之，这种奇奇怪怪的错误我也是第一次见，之前都没有见过，之前使用的时候都很正常。两个字有毒。

欢迎访问我的home：https://blog.csdn.net/weixin_43681796?spm=1000.2115.3001.5113 | https://www.zhihu.com/people/damon-51-73

posted @ 2020-11-06 17:36 Damon王阅读(1044) 评论(0) 收藏举报

刷新页面返回顶部

Loading

Damon王