Spark to ClickHouse 分布式故障排查：从 DNS 异常到竞态问题的根治实践

在分布式系统中，一个看似微小的网络抖动可能引发连锁故障。本文记录了一次 Spark 导入 ClickHouse 任务因 DNS 解析失败导致 Stage 重试，进而触发 Task 竞态、最终造成数据不一致的完整排查与修复过程。我们不仅定位了根因，还提出了高可用架构下的系统级改进方案，希望对从事大数据和微服务架构的工程师有所启发。

问题背景与任务架构

我们的 Spark Job 运行在 Yarn 集群上，核心功能是将 HDFS 上的 Parquet 数据导入 ClickHouse。整体流程如下：

读取与重分区：从 HDFS 读取 Parquet，通过 repartition 实现负载均衡，确保每个 Executor 分配到均匀的数据量。
本地转换与传输：每个 Executor 将分配到的数据写入本地 Parquet，再调用 ClickHouse-local 生成 ClickHouse 的本地 part，最后通过 SCP 将 part 传输到对应 ClickHouse 服务器的 detached 目录。
Driver 统一 Attach：当所有 Executor 的转换与传输 Job 成功后，Driver 一次性对所有 part 执行 ATTACH 操作，完成数据可见化。这种设计避免了 Executor 重试带来的不确定性，保证了分布式事务的原子性。

下图展示了整体数据流：

           ┌────────────────────────────┐
           │          Spark Driver       │
           └──────────────┬─────────────┘
                          │
                          │ 读取 HDFS parquet
                          v
                ┌──────────────────┐
                │   Dataset[Row]   │
                └─────────┬────────┘
                          │ repartition(...)
                          v
                ┌──────────────────┐
                │  toJavaRDD()     │
                └─────────┬────────┘
                          │ mapPartitions(ClickhouseSink)
                          v
        ┌────────────────────────────────────────────┐
        │              Spark Executors               │
        │            (每个 partition 一份)            │
        └──────────────┬─────────────────────────────┘
                       │
                       │ 1) 写 parquet-local
                       v
        /corp/data/.../parquet-local/.../partition_N/data_partition_N.parquet
                       │
                       │ 2) clickhouse-local 读 parquet 生成本地 part
                       v
        /corp/data/.../clickhouse-local/.../_local/{table}/{part}
                       │
                       │ 3) scp 到 ClickHouse server 的 detached
                       v
        /corp/data/.../clickhouse/store/.../detached/{part}
                       │
                       │ 4) 返回 AttachClickhouseInfo 给 Driver（collect）
                       v
           ┌────────────────────────────┐
           │          Spark Driver       │
           └──────────────┬─────────────┘
                          │
                          │ ALTER TABLE ... ATTACH PART
                          v
                 ClickHouse 表数据可见

排查过程：从 Driver 异常顺藤摸瓜

1. Driver 端的直接异常

Yarn 页面显示 ATTACH 操作失败：

26/01/21 15:03:21 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.RuntimeException: Failed to attach folder to clickhouse, host [rp506-2.iad7.prod.corp.com], clickhouseLocalFolders = [/corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_4/data/_local/oce_flow_pt1m_local/20260121_1_1_0_779054533,/corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_4/data/_local/oce_flow_pt1m_local/20260121_2_2_0_779054533]
at com.corp.storage.app.AppConverterLogic.attachClickhouse(AppConverterLogic.java:296)
at com.corp.storage.app.AppConverterLogic.lambda$execute$1(AppConverterLogic.java:138)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.corp.storage.app.AppConverterLogic.execute(AppConverterLogic.java:138)
at com.corp.storage.app.AppInsightsConverter.main(AppInsightsConverter.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)
Caused by: java.lang.RuntimeException: Retry failed.
at com.corp.storage.app.etl.core.utils.RetryUtil.retry(RetryUtil.java:37)
at com.corp.storage.app.etl.core.utils.RetryUtil.retry(RetryUtil.java:41)
at com.corp.storage.app.ch.ClickHouseOperation.attachToClickHouse(ClickHouseOperation.java:247)
at com.corp.storage.app.AppConverterLogic.attachClickhouse(AppConverterLogic.java:292)
... 19 more
Caused by: java.lang.RuntimeException: Clickhouse attach failed.
at com.corp.storage.app.ch.ClickHouseOperation.lambda$attachToClickHouse$1(ClickHouseOperation.java:266)
at com.corp.storage.app.etl.core.utils.RetryUtil.retry(RetryUtil.java:23)
... 22 more

。日志表明，在成功 ATTACH 多个 part 后，突然遇到一个 part 找不到的情况。此时数据转换与传输 Job 已成功完成，ClickHouse 集群状态正常，问题出现在 Driver 执行 ATTACH 的环节。

Driver 日志显示：

26/01/21 15:03:06 INFO ch.ClickHouseOperation: ClickHouse attach successfully.
26/01/21 15:03:06 INFO app.AppConverterLogic: AttachClickHouse, host: rp504-2.iad7.prod.corp.com, yarn local host: 10.72.1.146-oce1-spark-yarn-localssd-3.us-east4.prod.gcp.corp.com, clickhouseLocalFolders = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_2/data/_local/oce_flow_pt1m_local/20260121_1_1_0_779054531,/corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_2/data/_local/oce_flow_pt1m_local/20260121_2_2_0_779054531, clickhouseLocalPath = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_2, records = 440985
26/01/21 15:03:06 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_1_1_0_779054531'], host: rp504-2.iad7.prod.corp.com, detachedPath: /corp/data/nvme5n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:06 INFO ch.ClickHouseOperation: ClickHouse attach successfully.
26/01/21 15:03:06 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_2_2_0_779054531'], host: rp504-2.iad7.prod.corp.com, detachedPath: /corp/data/nvme5n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:06 INFO ch.ClickHouseOperation: ClickHouse attach successfully.
26/01/21 15:03:06 INFO app.AppConverterLogic: AttachClickHouse, host: rp505-1.iad7.prod.corp.com, yarn local host: 10.72.1.146-oce1-spark-yarn-localssd-3.us-east4.prod.gcp.corp.com, clickhouseLocalFolders = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_3/data/_local/oce_flow_pt1m_local/20260121_1_1_0_779054532,/corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_3/data/_local/oce_flow_pt1m_local/20260121_2_2_0_779054532, clickhouseLocalPath = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_3, records = 440986
26/01/21 15:03:06 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_1_1_0_779054532'], host: rp505-1.iad7.prod.corp.com, detachedPath: /corp/data/nvme6n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:06 INFO ch.ClickHouseOperation: ClickHouse attach successfully.
26/01/21 15:03:06 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_2_2_0_779054532'], host: rp505-1.iad7.prod.corp.com, detachedPath: /corp/data/nvme6n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:06 INFO ch.ClickHouseOperation: ClickHouse attach successfully.
26/01/21 15:03:06 INFO app.AppConverterLogic: AttachClickHouse, host: rp506-2.iad7.prod.corp.com, yarn local host: 10.72.1.146-oce1-spark-yarn-localssd-3.us-east4.prod.gcp.corp.com, clickhouseLocalFolders = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_4/data/_local/oce_flow_pt1m_local/20260121_1_1_0_779054533,/corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_4/data/_local/oce_flow_pt1m_local/20260121_2_2_0_779054533, clickhouseLocalPath = /corp/data/ab-etl-workspace/clickhouse-local/default/oce_flow_pt1m_local/2026/01/21/14/59/fcc00f88-564c-427a-917c-8499a52d6c99/partition_4, records = 440988
26/01/21 15:03:06 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_1_1_0_779054533'], host: rp506-2.iad7.prod.corp.com, detachedPath: /corp/data/nvme7n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:06 WARN utils.RetryUtil: returnTime = 1, retry exception: java.lang.RuntimeException: Clickhouse attach failed.
26/01/21 15:03:07 INFO scheduler.TaskSetManager: Ignoring task-finished event for 6.0 in stage 2.1 because task 6 has already completed successfully
26/01/21 15:03:08 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_1_1_0_779054533'], host: rp506-2.iad7.prod.corp.com, detachedPath: /corp/data/nvme7n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:08 WARN utils.RetryUtil: returnTime = 2, retry exception: java.lang.RuntimeException: Clickhouse attach failed.
26/01/21 15:03:12 INFO ch.ClickHouseOperation: Attach command: [ALTER TABLE oce_flow_pt1m_local ATTACH PART '20260121_1_1_0_779054533'], host: rp506-2.iad7.prod.corp.com, detachedPath: /corp/data/nvme7n1/clickhouse/store/57a/57aa2c6f-915b-4a4f-a080-ef8975632763/detached/
26/01/21 15:03:12 WARN utils.RetryUtil: returnTime = 3, retry exception: java.lang.RuntimeException: Clickhouse attach failed.

。我们注意到，Driver 从 Executor 上报的信息中获知 part 已就绪，但实际 ATTACH 时 ClickHouse 却返回“Part not exists”。这暗示 part 在传输后、ATTACH 前被意外删除或覆盖。

2. Task 1.0 in Stage 2.0 失败：DNS 异常引发蝴蝶效应

进一步查看 Driver 日志，第一条明显异常是 FetchFailedException，堆栈指向 UnknownHostException：

26/01/21 15:02:09 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 2.0 (TID 23, rp408-25a.iad6.prod.corp.com, executor 10): FetchFailed(BlockManagerId(4, oce1-spark-yarn-localssd-1.us-east4.prod.gcp.corp.com, 7337, None), shuffleId=0, mapId=6, reduceId=1, message=
org.apache.spark.shuffle.FetchFailedException: Failed to connect to oce1-spark-yarn-localssd-1.us-east4.prod.gcp.corp.com:7337
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:523)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:454)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:61)
.....
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
at com.corp.storage.app.transform.engine.sink.IRddSinkTemplate$1.hasNext(IRddSinkTemplate.java:40)
at com.corp.storage.app.etl.core.utils.ParquetUtil.writeRecords(ParquetUtil.java:47)
at com.corp.storage.app.etl.core.utils.ParquetUtil.write(ParquetUtil.java:29)
at com.corp.storage.app.transform.engine.sink.Rdd2ParquetSinkTemplate.sink(Rdd2ParquetSinkTemplate.java:26)
at com.corp.storage.app.excutors.ClickhouseSink.writeLocalParquetByRdd(ClickhouseSink.java:180)
at com.corp.storage.app.excutors.ClickhouseSink.call(ClickhouseSink.java:89)
at com.corp.storage.app.excutors.ClickhouseSink.call(ClickhouseSink.java:35)
....
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to connect to oce1-spark-yarn-localssd-1.us-east4.prod.gcp.corp.com:7337
...
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Caused by: java.net.UnknownHostException: oce1-spark-yarn-localssd-1.us-east4.prod.gcp.corp.com
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
...
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
... 2 more
)
26/01/21 15:02:09 INFO scheduler.DAGScheduler: Marking ResultStage 2 (collect at AppConverterLogic.java:106) as failed due to a fetch failure from ShuffleMapStage 1 (toJavaRDD at AppConverterLogic.java:106)

。这说明 Executor 10 在拉取 shuffle 数据时，无法解析另一台主机的主机名。我们的 Yarn 集群是混合部署（本地数据中心 + GCP），DNS 抖动偶有发生。

这个 DNS 异常导致 Stage 2.0 中 Task 1.0 失败，进而触发 Stage 级别重试（Stage 2.1）。Spark 不会 Kill 正在运行的 Stage 2.0 Task，而是让两个 Stage Attempt 同时运行。这为后续竞态问题埋下了伏笔。

⚔️ 竞态分析：Stage 重试下的多方冲突

1. Stage 2.0 重试机制的关键细节

Spark 判断 Job 完成的标准是“所有 Task 成功”，而非“所有 Stage Attempt 结束”。这意味着：

Stage 2.0 中失败的 Task 1.0 在 Stage 2.1 中重试成功后，Job 即可标记完成，即使 Stage 2.1 的其他 Task 仍在执行。
Stage 2.1 启动时，Stage 2.0 的剩余 Task 仍在运行，两者操作相同的远程资源（ClickHouse detached 目录）和本地资源（临时文件），极易产生竞态。

Stage 依赖关系图：

Driver: AppConverterLogic.execute()
  ┌───────────────────────────────────────────────────────────────────────┐
  │ Job 0                                                                  │
  │   ResultStage 0  (load at AppConverterLogic.java:196)                  │
  │   作用：spark.read.format(\"parquet\").load(hdfsPaths...) 的读取准备工作 │
  │        （列文件/读 footer/合并 schema 等）                              │
  └───────────────────────────────────────────────────────────────────────┘
                     |
                     |  Dataset parquetData
                     v
  ┌───────────────────────────────────────────────────────────────────────┐
  │ Job 1                                                                  │
  │   Stage 1: ShuffleMapStage 1                                           │
  │     触发点：readSourceParquet() 里的 repartition(partitionNum)         │
  │     作用：把输入数据按新的分区数洗牌，生成 shuffle blocks              │
  │                                                                       │
  │                 shuffle dependency                                     │
  │                 (shuffleId = 0, mapId/reduceId in logs)                │
  │                        │                                                │
  │                        v                                                │
  │   Stage 2: ResultStage 2  (collect at AppConverterLogic.java:106)      │
  │     触发点：parquetData.toJavaRDD().mapPartitions(chSink).collect()     │
  │     作用：每个 partition 执行 ClickhouseSink                            │
  │          - 写本地 parquet                                                │
  │          - clickhouse-local 生成本地 part                                │
  │          - scp 到 ClickHouse detached/                                   │
  │     输出：collect() 返回 List 给 driver            │
  └───────────────────────────────────────────────────────────────────────┘
                     |
                     |  driver 收到 attachClickhouseInfos
                     v
  ┌───────────────────────────────────────────────────────────────────────┐
  │ Driver side (不属于 Spark stage)                                       │
  │   attachClickhouseInfos.forEach(attachClickhouse(...))                 │
  │   作用：对每个 part 执行 ALTER TABLE ... ATTACH PART                    │
  └───────────────────────────────────────────────────────────────────────┘

2. Task 3.0 在 Stage 2 两个 Attempt 中的“两方竞态”

Task 3.0 在 Stage 2.0 和 Stage 2.1 中恰好被调度到同一台物理机。当 Stage 2.0 的 Task 3.0 先完成 SCP 后，它删除了本地的临时文件。此时 Stage 2.1 的 Task 3.0 正在执行 SCP，发现本地文件已被删除，抛出 FileNotFoundException。这个异常很容易被误判为业务逻辑问题，但实际上它只是 DNS 异常的“衍生灾害”。

3. Task 4.0 在 Stage 2 两个 Attempt 中的“三方竞态”

更严重的是，Task 4.0 在 Stage 2.0 中成功将 part 传输到 ClickHouse 服务器 A 的 detached 目录。随后 Task 4.0 在 Stage 2.1 中也成功传输了相同的 part（因为 Stage 2.0 的 Task 4.0 并未被 Kill）。当 Stage 2.1 的 Task 4.0 完成 SCP 后，它执行了 清理操作，删除了 Stage 2.0 传输的 part 文件！最终 Driver 在 ATTACH 时，只能找到 Stage 2.1 的 part（如果后者还未被清理），或者什么都找不到。

这正是 Driver 端 ATTACH 失败的根本原因：Stage 重试导致多个 Attempt 的 Task 操作同一份远程资源，后执行的 Task 覆盖或删除了先执行的 Task 的结果。

️ 解决方案：把竞态窗口“关上”

我们提出了以下系统架构层面的改进方案，以消除竞态窗口：

DNS 兜底策略：在 Executor 启动时，预解析所有 ClickHouse 主机名并缓存 IP，避免运行时 DNS 抖动。同时配置本地 hosts 文件作为 fallback。
Detached 目录原子发布：Executor 不再直接 SCP 到 ClickHouse 的 detached 目录，而是先传输到一个临时目录，然后通过原子重命名（rename）操作移动到 detached 目录。这样即使多个 Attempt 同时写入，也不会相互覆盖。
重试隔离：本地路径 + 远端命名：每个 Task 的本地临时文件路径和远端 part 文件名都包含 Stage Attempt ID（例如 part_${stageAttemptId}_${taskId}），确保不同 Attempt 的 Task 操作互不冲突。Driver 在 ATTACH 时，只选择最新 Attempt 的 part。
Job 完成条件优化：在 Spark 层面，我们通过自定义监听器，在 Job 完成前确保所有旧 Attempt 的 Task 已被 Kill，避免残留 Task 干扰。

以下是我们最终实现的代码片段（仅供示意）：

attemptKey = s_sa_ta

[AFFILIATE_SLOT_1] 如果你也在构建高可用的数据管道，建议关注 Spark 重试机制与外部系统的交互设计，避免类似竞态问题。

总结：分布式排查的方法论

本次故障排查让我们深刻体会到：

建立因果时间线：从 Driver 异常出发，顺藤摸瓜找到 Executor 日志，再回溯到 Stage 重试和 DNS 异常，最终定位竞态。
识别噪声异常：Task 3.0 的 FileNotFoundException 看似严重，实则是 DNS 异常的“副产品”，与最终失败无关。
修复根因 + 增强韧性：既要解决 DNS 抖动问题，也要优化代码以优雅处理重试场景，防止单一故障演变成系统雪崩。

整个排查过程涉及多个组件（Spark、Yarn、ClickHouse、DNS），需要综合运用日志分析、源码阅读、分布式系统知识。希望本文的案例能为你排查类似问题提供参考。

[AFFILIATE_SLOT_2] 欢迎在评论区分享你的分布式系统故障排查经验，一起探讨高可用架构的最佳实践。

发表于 2026-07-21 16:28 jzssuanfa 阅读(13) 评论(0) 收藏举报

刷新页面返回顶部