Retrying task after failure: Waiting for lock. org.apache.iceberg.hive.HiveTableOperations$WaitingForLockException: Waiting for lock.

执行spark任务居然碰到了hive中表的被锁了,导致写数据时失败,这个还真是头一次碰到过这回事。

错误信息如下:

22/09/04 21:02:28 WARN Tasks: Retrying task after failure: Waiting for lock.
org.apache.iceberg.hive.HiveTableOperations$WaitingForLockException: Waiting for lock.
        at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$9(HiveTableOperations.java:460)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
        at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:454)
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226)
        at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127)
        at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:304)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
        at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282)
        at org.apache.iceberg.BaseReplacePartitions.commit(BaseReplacePartitions.java:26)
        at org.apache.iceberg.spark.source.Writer.commitOperation(Writer.java:144)
        at org.apache.iceberg.spark.source.Writer.replacePartitions(Writer.java:181)
        at org.apache.iceberg.spark.source.Writer.commit(Writer.java:119)
        at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:76)
 

跑到hive的元数据库中查询时,在表HIVE_LOCKS中找到了一些信息。
image

根据出错信息,根据hl_table找到相关的表,查看最近的心跳时间字段,hl_last_heartbeat。
删除对应的记录即可。

delete  from HIVE_LOCKS where hl_table = '出错的表名';
posted @ 2022-09-04 21:18  硅谷工具人  阅读(485)  评论(0)    收藏  举报
成功之道,在于每个人生阶段都要有不同的目标,并且通过努力实现自己的目标,毕竟人生不过百年! 所有奋斗的意义在于为个人目标实现和提升家庭幸福,同时能推进社会进步和国家目标! 正如古人讲的正心诚意格物致知,修身齐家治国平天下。