12 2019 档案

hivesql迁移spark2.4 cannot resolve '(ctime >= start_time) due to data type mismatch: differing types in '(ctime` >= `start_time`)'(timestamp and bigint).; line 99 pos 10
摘要：hivesql 一些默认类型转换。但spark需要手动强转 比如以下部分: SELECT 1 as id, avid as avid, mid as mid, TRIM(LOWER(tag)) as tag, ctime as ctime FROM archive.dws_archive_daily 阅读全文
posted @ 2019-12-31 14:32 songchaolin 阅读(1672) 评论(0) 推荐(0) 编辑
sparksql报错 cannot resolve 'coalesce(d.`field`, map(NULL, NULL))' due to data type mismatch: input to function coalesce should all be the same type
摘要：hivesql可运行的sql 在spark上运行报错如下 Error in query: cannot resolve 'coalesce(d.`subarea_duration_l30`, map(NULL, NULL))' due to data type mismatch: input to 阅读全文
posted @ 2019-12-26 17:30 songchaolin 阅读(1329) 评论(0) 推荐(0) 编辑
hivesql 迁移spark3.0 sparksql报错如Cannot safely cast '字段':StringType to IntegerType的问题
摘要：一 问题 hivesql可以正常运行，spark3.0运行报错如图 spark3.0配置 查看源码新增一个 val STORE_ASSIGNMENT_POLICY = buildConf("spark.sql.storeAssignmentPolicy") .doc("When inserting 阅读全文
posted @ 2019-12-25 19:58 songchaolin 阅读(6371) 评论(0) 推荐(0) 编辑
sparksql 报错Container killed by YARN for exceeding memory limits. xGB of x GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling
摘要：对此 提高了对外内存 spark.executor.memoryOverhead = 4096m 重新执行sql 改报下面的错误 19/12/25 15:49:02 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from bigd 阅读全文
posted @ 2019-12-25 18:55 songchaolin 阅读(3292) 评论(0) 推荐(1) 编辑
hiveSql 迁移spark2.4时报错Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause
摘要：hivesql对语法检查较弱 像下面的语法 hive是可以通过的 partition by 后没有跟order by row_number() over(partition by buvid,version_code,app_id) as rn 原因看下hive 源码(hive 已经做了补充) sp 阅读全文
posted @ 2019-12-25 16:08 songchaolin 阅读(1146) 评论(0) 推荐(0) 编辑
spark 执行报错 java.io.EOFException: Premature EOF from inputStream
摘要：使用spark2.4跟spark2.3 做替代公司现有的hive选项。 跑个别任务spark有以下错误 java.io.EOFException: Premature EOF from inputStream at com.hadoop.compression.lzo.LzopInputStream 阅读全文
posted @ 2019-12-23 19:42 songchaolin 阅读(2499) 评论(0) 推荐(0) 编辑
记录一次线上yarn RM频繁切换的故障
摘要：周末一大早被报警惊醒，rm频繁切换 急急忙忙排查 看到两处错误日志 错误信息1 ervation <memory:0, vCores:0> 2019-12-21 11:51:57,781 FATAL org.apache.hadoop.yarn.server.resourcemanager.Reso 阅读全文
posted @ 2019-12-21 15:13 songchaolin 阅读(1075) 评论(0) 推荐(0) 编辑
编译+远程调试spark
摘要：一 编译 以spark2.4 hadoop2.8.4为例 1，spark 项目根pom文件修改 pom文件新增 <profile> <id>hadoop-2.8</id> <properties> <hadoop.version>2.8.4</hadoop.version> </properties 阅读全文
posted @ 2019-12-12 11:54 songchaolin 阅读(431) 评论(0) 推荐(0) 编辑