bthread_join 内存可见性问题复现 Demo

测试环境：

cpu: aarch 64 

os: openEuler 22.03 (LTS-SP4)

代码准备：

/**
 * bthread_join 内存可见性问题复现 Demo
 *
 * 原理：bthread 内写入大量数据到共享结构，主线程 bthread_join 后立即读取。
 * 在 ARM (aarch64) 弱内存序下，由于 bthread_join 缺少 acquire fence，
 * joining 线程可能读到过期缓存数据（stale data）。
 *
 * 提高复现概率的策略：
 * 1. 使用多个 CPU 核心，确保 bthread 和主线程在不同物理核上执行
 * 2. 大量迭代，统计出错次数
 * 3. 写入足够多的数据，增大 store buffer / invalidate queue 延迟窗口
 * 4. join 后立即读取，缩小时间窗口
 * 5. 多线程并发执行，增加竞争
 */

#include <bthread/bthread.h>
#include <butil/logging.h>
#include <atomic>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <vector>
#include <thread>
#include "gflags/gflags.h"

// 魔数，用于检测写入是否可见
static constexpr uint64_t MAGIC_VALUE = 0xDEADBEEFCAFEBABEULL;
static constexpr uint64_t JEMALLOC_JUNK = 0xa5a5a5a5a5a5a5a5ULL;

// 每次迭代中 bthread 写入的数据量（越大，store buffer 排空延迟越大）
static constexpr int DATA_SIZE = 64;

struct SharedData {
    // 用 vector 模拟生产环境的 emplace_back 场景
    std::vector<uint64_t> values;
    // 额外的填充数据，增大写入量以扩大 stale read 窗口
    uint64_t padding[DATA_SIZE];
};

static void* bthread_worker(void* arg) {
    SharedData* data = reinterpret_cast<SharedData*>(arg);

    // 先写填充数据（撑大 store buffer）
    for (int i = 0; i < DATA_SIZE; ++i) {
        data->padding[i] = MAGIC_VALUE;
    }

    // 最后写入 vector（模拟生产环境的 emplace_back）
    data->values.emplace_back(MAGIC_VALUE);

    return nullptr;
}

struct Stats {
    std::atomic<uint64_t> total_iters{0};
    std::atomic<uint64_t> stale_reads{0};
    std::atomic<uint64_t> vector_stale{0};
    std::atomic<uint64_t> padding_stale{0};
};

static void run_test(Stats* stats, int iterations) {
    uint64_t local_stale = 0;
    uint64_t local_vec_stale = 0;
    uint64_t local_pad_stale = 0;

    for (int i = 0; i < iterations; ++i) {
        // 每次迭代使用新的 SharedData，确保内存是"新鲜"的
        // 这样更容易触发 jemalloc junk 填充 -> stale read 的场景
        SharedData* data = new SharedData();
        data->values.reserve(1);  // 预分配，避免 emplace_back 时 realloc

        bthread_t tid;
        int rc = bthread_start_background(&tid, nullptr, bthread_worker, data);
        if (rc != 0) {
            fprintf(stderr, "bthread_start_background failed: %d\n", rc);
            delete data;
            continue;
        }

        // bthread_join 返回后，按语义应该能看到 bthread 内所有写入
        bthread_join(tid, nullptr);

        // 立即检查数据可见性（不要有任何额外操作，缩小窗口）
        bool has_stale = false;

        // 检查填充数据
        for (int j = 0; j < DATA_SIZE; ++j) {
            if (data->padding[j] != MAGIC_VALUE) {
                local_pad_stale++;
                has_stale = true;
                fprintf(stderr, "[%s:%d][iter %d] padding[%d] stale read: got 0x%lx%s\n",
                        __FILE__, __LINE__, i, j, data->padding[j],
                        (data->padding[j] == JEMALLOC_JUNK) ? " (jemalloc junk!)" : "");
                break;  // 只报告第一个
            }
        }

        // 检查 vector 数据
        if (data->values.empty() || data->values[0] != MAGIC_VALUE) {
            local_vec_stale++;
            has_stale = true;
            if (data->values.empty()) {
                fprintf(stderr, "[%s:%d][iter %d] vector is EMPTY after join!\n", __FILE__, __LINE__, i);
            } else {
                uint64_t got = data->values[0];
                fprintf(stderr, "[%s:%d][iter %d] vector stale read: got 0x%lx, expected 0x%lx%s\n",
                        __FILE__, __LINE__, i, got, MAGIC_VALUE,
                        (got == JEMALLOC_JUNK) ? " (jemalloc junk!)" : "");
            }
        }

        if (has_stale) {
            local_stale++;
        }

        delete data;
    }

    stats->total_iters.fetch_add(iterations, std::memory_order_relaxed);
    stats->stale_reads.fetch_add(local_stale, std::memory_order_relaxed);
    stats->vector_stale.fetch_add(local_vec_stale, std::memory_order_relaxed);
    stats->padding_stale.fetch_add(local_pad_stale, std::memory_order_relaxed);
}

int main(int argc, char* argv[]) {
    google::ParseCommandLineFlags(&argc, &argv, true);
    int num_threads = 4;        // 并发线程数
    int iterations = 1000000;   // 每线程迭代次数

    if (argc > 1) num_threads = atoi(argv[1]);
    if (argc > 2) iterations = atoi(argv[2]);

    printf("=== bthread_join Memory Visibility Stress Test ===\n");
    printf("Threads: %d, Iterations per thread: %d\n", num_threads, iterations);
    printf("Total iterations: %d\n", num_threads * iterations);
    printf("Platform: ");
#if defined(__aarch64__)
    printf("aarch64 (ARM64) - weak memory model, expect stale reads\n");
#elif defined(__x86_64__)
    printf("x86_64 - TSO strong memory model, stale reads unlikely\n");
#else
    printf("unknown\n");
#endif
    printf("Running...\n\n");

    Stats stats;
    std::vector<std::thread> threads;

    for (int i = 0; i < num_threads; ++i) {
        threads.emplace_back(run_test, &stats, iterations);
    }

    for (auto& t : threads) {
        t.join();
    }

    printf("\n=== Results ===\n");
    printf("Total iterations:     %lu\n", stats.total_iters.load());
    printf("Stale read incidents: %lu\n", stats.stale_reads.load());
    printf("  - vector stale:     %lu\n", stats.vector_stale.load());
    printf("  - padding stale:    %lu\n", stats.padding_stale.load());

    if (stats.stale_reads.load() > 0) {
        double rate = 100.0 * stats.stale_reads.load() / stats.total_iters.load();
        printf("\n*** STALE READS DETECTED! Rate: %.6f%% ***\n", rate);
        printf("This confirms bthread_join lacks acquire fence on this platform.\n");
    } else {
        printf("\nNo stale reads detected in this run.\n");
#if defined(__aarch64__)
        printf("Try increasing iterations or thread count for higher probability.\n");
#elif defined(__x86_64__)
        printf("Expected on x86 (TSO) - hardware provides strong ordering.\n");
#endif
    }

    return stats.stale_reads.load() > 0 ? 1 : 0;
}

 

brpc中bthread_join修复：

--- a/src/bthread/task_group.cpp
+++ b/src/bthread/task_group.cpp

 namespace bthread {
@@ -594,6 +596,14 @@ int TaskGroup::join(bthread_t tid, void** return_value) {
             return errno;
         }
     }
+    if (brpc::FLAGS_brpc_bthread_join_add_fence) {
+#if defined(__aarch64__) || defined(__arm__)
+        // On ARM's weak memory model, ensure all memory writes made by the
+        // joined bthread are visible to the joining thread after join returns.
+        // This matches the semantic guarantee provided by pthread_join().
+        butil::atomic_thread_fence(butil::memory_order_acquire);
+#endif
+    }
     if (return_value) {
         *return_value = NULL;
     }

 

运行结果：

未开启gflag时运行结果如下：

=== bthread_join Memory Visibility Stress Test ===
Threads: 4, Iterations per thread: 1000000
Total iterations: 4000000
Platform: aarch64 (ARM64) - weak memory model, expect stale reads
Running...

[bthread_join_visibility_demo.cpp:92][iter 171378] padding[13] stale read: got 0x0
[bthread_join_visibility_demo.cpp:92][iter 557221] padding[0] stale read: got 0x0

=== Results ===
Total iterations: 4000000
Stale read incidents: 2
- vector stale: 0
- padding stale: 2

*** STALE READS DETECTED! Rate: 0.000050% ***
This confirms bthread_join lacks acquire fence on this platform.

开启gflag后运行效果如下：

=== bthread_join Memory Visibility Stress Test ===
Threads: 4, Iterations per thread: 1000000
Total iterations: 4000000
Platform: aarch64 (ARM64) - weak memory model, expect stale reads
Running...

=== Results ===
Total iterations: 4000000
Stale read incidents: 0
- vector stale: 0
- padding stale: 0

No stale reads detected in this run.

结果分析

对比

| 场景 | stale read | 说明 | |------|-----------|------|

| FLAGS_brpc_bthread_join_add_fence=false | 2 次（0.00005%） | padding[0]、padding[13] 读到 0x0 |

| FLAGS_brpc_bthread_join_add_fence=true | 0 次 | 400万次全部正确 |

分析

开启 gflag 后，TaskGroup::join() 在 while 循环退出后执行了 atomic_thread_fence(memory_order_acquire)（第 604 行），对应 ARM 指令 DMB ISHLD，强制 CPU 立即清空 invalidate queue，确保 bthread 写入的所有数据对 joining 线程可见。

关闭时，while 循环通过普通读 *m->version_butex 退出，无 acquire fence，invalidate queue 中可能还有未处理的失效消息，导致主线程读到过期缓存行（构造时的 0x0）。

结论

这组对比实验直接证明了：

1. 根因确认：bthread_join 缺少 acquire fence 是 ARM 上 stale read 的根因
2. 修复有效：一条 DMB ISHLD 即可彻底消除问题