2019年12月
hivesql 迁移spark3.0 sparksql报错如Cannot safely cast '字段':StringType to IntegerType的问题
摘要： 一 问题 hivesql可以正常运行，spark3.0运行报错如图 spark3.0配置 查看源码新增一个 val STORE_ASSIGNMENT_POLICY = buildConf("spark.sql.storeAssignmentPolicy") .doc("When inserting 阅读全文
posted @ 2019-12-25 19:58 songchaolin 阅读(6187) 评论(0) 推荐(0) 编辑
sparksql 报错Container killed by YARN for exceeding memory limits. xGB of x GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling
摘要： 对此 提高了对外内存 spark.executor.memoryOverhead = 4096m 重新执行sql 改报下面的错误 19/12/25 15:49:02 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from bigd 阅读全文
posted @ 2019-12-25 18:55 songchaolin 阅读(3244) 评论(0) 推荐(1) 编辑
hiveSql 迁移spark2.4时报错Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause
摘要： hivesql对语法检查较弱 像下面的语法 hive是可以通过的 partition by 后没有跟order by row_number() over(partition by buvid,version_code,app_id) as rn 原因看下hive 源码(hive 已经做了补充) sp 阅读全文
posted @ 2019-12-25 16:08 songchaolin 阅读(1137) 评论(0) 推荐(0) 编辑