Lab 2.5: Concurrency Debugging

Objectives

In the earlier labs we kept practicing "how to write concurrent code correctly". This lab flips direction — we are facing code that is already written, and it contains bugs. Your task is not to build from scratch, but to use tools and a systematic method to locate the problem, understand the root cause, fix it, and verify the fix with a regression run.

This lab provides 5 "deliberately broken concurrent programs"; each program contains one concurrency defect of a known type — a data race, a lost wakeup, a deadlock, a use-after-free, and a false sharing performance trap. Without peeking at the answers, you need to walk through the complete debugging loop of "locate → hypothesize → verify → fix". The code reuses patterns you already know from Labs 0–2 (thread pools, queues, atomic counters), so the comprehension cost is low and you can put all your attention on the debugging process itself.

Prerequisites

Before starting, make sure you have finished reading the following sections:

• ch08-01: Debugging techniques for concurrent programs — TSan, helgrind, custom logging
• ch08-02: Concurrency performance testing and benchmarking — perf, performance analysis methods
• Labs 0–2: understanding the basic structure of JoiningThread, BoundedBlockingQueue, and SpscRingBuffer

Environment Setup

The core of this lab is toolchain configuration. You need the following tools:

• TSan: compile with -fsanitize=thread -g on GCC/Clang
• helgrind: part of Valgrind, valgrind --tool=helgrind ./program
• perf: the Linux profiling tool, perf stat and perf record

Install Valgrind (if not yet installed):

# Ubuntu/Debian
sudo apt install valgrind linux-perf

# WSL2
sudo apt install valgrind
# perf may need extra steps; see the WSL2 documentation

Debugging Methodology

Before tackling the bug programs one by one, let's establish a unified debugging workflow. Whenever you hit a concurrency problem, proceed through these steps:

Step 1: Confirm it really is a concurrency problem. Run the same logic single-threaded; if the result is correct, the defect really was introduced by concurrency.

Step 2: Shrink the reproduction. Reduce the thread count, data volume, and number of iterations until you find the minimal reproduction path. The smaller the reproducing code, the easier it is to locate.

Step 3: Pick the right tool. TSan for data races, helgrind or valgrind --tool=drd for deadlocks, perf stat for performance anomalies.

Step 4: Interpret the report. What do "previous write" and "current read" in a TSan report mean? How do you read the lock-order graph in a helgrind report?

Step 5: Regression after the fix. Not just "it ran through once" — run the complete test suite under TSan and confirm the problem is gone for good.

Bug 1: Data Race on Shared Counter

Symptom

Multiple threads modify the same int counter without locking, and the final result does not equal the expected value. Run it several times and every run gives a different result.

The Buggy Code

Expand codeCollapse32 lines

// bug1_data_race.cpp
#include <thread>
#include <vector>
#include <iostream>

int counter = 0;  // note: a plain non-atomic int

void increment(int times)
{
    for (int i = 0; i < times; ++i) {
        ++counter;  // modified by multiple threads at once → data race
    }
}

int main()
{
    const int kThreads = 8;
    const int kTimes = 1000000;
    std::vector<std::thread> threads;

    for (int i = 0; i < kThreads; ++i) {
        threads.emplace_back(increment, kTimes);
    }

    for (auto& t : threads) {
        t.join();
    }

    std::cout << "Expected: " << kThreads * kTimes << "\n";
    std::cout << "Actual:   " << counter << "\n";
    return 0;
}

Debugging Tasks

1. Run the program and record how far the actual output is from the expected value
2. Run it under TSan and interpret where the report says the data race is
3. Fix the code (replace int with std::atomic<int>)
4. Pick an appropriate memory order (hint: relaxed is enough for pure counting)
5. Run a TSan regression to verify the fix

Verification

After the fix, run it 10 times in a row; every result should equal kThreads * kTimes (8,000,000). TSan should no longer report anything.

Bug 2: Lost Wakeup

Symptom

The producer calls notify_one() before the consumer has entered wait(), so the consumer blocks forever. The program hangs.

The Buggy Code

Expand codeCollapse38 lines

// bug2_lost_wakeup.cpp
#include <mutex>
#include <condition_variable>
#include <thread>
#include <iostream>

std::mutex mtx;
std::condition_variable cv;
bool ready = false;

void consumer()
{
    std::unique_lock<std::mutex> lock(mtx);
    // Bug: wait without a predicate
    // If notify already happened before wait, wait blocks forever
    cv.wait(lock);
    std::cout << "Consumer: got the signal\n";
}

void producer()
{
    // notify happens before the consumer enters wait
    std::lock_guard<std::mutex> lock(mtx);
    ready = true;
    cv.notify_one();
    std::cout << "Producer: sent signal\n";
}

int main()
{
    // This scheduling order can trigger a lost wakeup
    std::thread p(producer);
    std::thread c(consumer);

    p.join();
    c.join();  // may block forever
    return 0;
}

Debugging Tasks

1. Run the program several times and observe whether it always hangs (it depends on thread scheduling)
2. Use timeout logging to aid the diagnosis: give wait a wait_for timeout and print a log line when the timeout expires
3. Fix the code: change cv.wait(lock) to cv.wait(lock, [&]{ return ready; })
4. Explain why waiting on a predicate solves both spurious wakeups and lost wakeups

Verification

After the fix the program should exit normally within 1 second. Try different thread startup orders and confirm they all run correctly.

Bug 3: Deadlock from Lock Ordering

Symptom

Two threads acquire two mutexes in opposite orders, and a particular schedule produces a deadlock. The program hangs.

The Buggy Code

Expand codeCollapse36 lines

// bug3_deadlock.cpp
#include <mutex>
#include <thread>
#include <iostream>

std::mutex mutex_a;
std::mutex mutex_b;

void task1()
{
    std::lock_guard<std::mutex> lock_a(mutex_a);  // lock A first
    std::cout << "Task1: locked A\n";
    std::this_thread::sleep_for(std::chrono::milliseconds(10));

    std::lock_guard<std::mutex> lock_b(mutex_b);  // then lock B
    std::cout << "Task1: locked B\n";
}

void task2()
{
    std::lock_guard<std::mutex> lock_b(mutex_b);  // lock B first
    std::cout << "Task2: locked B\n";
    std::this_thread::sleep_for(std::chrono::milliseconds(10));

    std::lock_guard<std::mutex> lock_a(mutex_a);  // then lock A
    std::cout << "Task2: locked A\n";
}

int main()
{
    std::thread t1(task1);
    std::thread t2(task2);
    t1.join();
    t2.join();
    return 0;
}

Debugging Tasks

1. Run the program several times and observe whether it occasionally hangs (a deadlock requires a specific scheduling order)
2. Run it under helgrind: valgrind --tool=helgrind ./bug3_deadlock, and interpret the lock-order conflict report
3. Use std::scoped_lock(mutex_a, mutex_b) to acquire both locks at once and eliminate the ordering problem
4. Or make both threads use the same locking order (A first, then B)

Verification

After the fix, 100 consecutive runs should never hang. helgrind should no longer report lock-order conflicts.

Bug 4: Use-After-Free in Detached Thread

Symptom

After the thread is detached, it keeps accessing a local variable that has already been destroyed. The program may crash, or it may print garbage values — the behavior depends entirely on scheduling timing.

The Buggy Code

Expand codeCollapse25 lines

// bug4_use_after_free.cpp
#include <thread>
#include <string>
#include <iostream>

void start_background_task()
{
    std::string message = "Hello from background";

    std::thread t([&message]() {
        // Bug: after detach, message may already be destroyed
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
        std::cout << message << "\n";  // use-after-free!
    });
    t.detach();
    // the function returns, message is destroyed
}

int main()
{
    start_background_task();
    // the main thread exits while the detached thread may still be accessing the destroyed message
    std::this_thread::sleep_for(std::chrono::milliseconds(200));
    return 0;
}

Debugging Tasks

1. Run the program several times — sometimes it prints normally, sometimes it prints garbage, sometimes it segfaults
2. Run it under TSan and look at the use-after-free report (TSan can detect accesses to freed memory)
3. Fix the code: capture by value instead of by reference, [message]() { ... }, so the thread owns its own copy
4. Or replace detach with a JoiningThread to ensure the thread finishes before the variables are destroyed

Verification

After the fix, 50 consecutive runs should all print "Hello from background" normally. TSan should no longer report anything.

Bug 5: False Sharing Performance Trap

Symptom

Two threads each modify atomic variables sitting at adjacent memory locations, and performance falls far below expectations. Functionally everything is correct, yet throughput is even lower than that of the single-threaded version.

The Buggy Code

Expand codeCollapse42 lines

// bug5_false_sharing.cpp
#include <atomic>
#include <thread>
#include <iostream>
#include <chrono>

struct Counters {
    std::atomic<int> a{0};  // two atomics packed side by side
    std::atomic<int> b{0};  // possibly on the same cache line
};

int main()
{
    Counters counters;
    const int kIterations = 50000000;

    auto start = std::chrono::steady_clock::now();

    std::thread t1([&]() {
        for (int i = 0; i < kIterations; ++i) {
            counters.a.fetch_add(1, std::memory_order_relaxed);
        }
    });

    std::thread t2([&]() {
        for (int i = 0; i < kIterations; ++i) {
            counters.b.fetch_add(1, std::memory_order_relaxed);
        }
    });

    t1.join();
    t2.join();

    auto elapsed = std::chrono::steady_clock::now() - start;
    auto ms = std::chrono::duration_cast<
        std::chrono::milliseconds>(elapsed).count();

    std::cout << "Time: " << ms << " ms\n";
    std::cout << "a = " << counters.a.load() << "\n";
    std::cout << "b = " << counters.b.load() << "\n";
    return 0;
}

Debugging Tasks

1. Run this version first and record the elapsed time
2. Fix it: add cache line padding between the two atomics (alignas(64) or manual padding)
3. Use perf stat to observe how the cache miss count changes before and after the fix
4. Compare the elapsed times before and after the fix and compute the speedup

The fixed structure should look something like this:

struct Counters {
    alignas(64) std::atomic<int> a{0};
    alignas(64) std::atomic<int> b{0};
};

Verification

After the fix, the elapsed time should be 2-5x faster than before (depending on the CPU architecture). Watch the cache-misses metric in perf stat and it should drop significantly.

Self-Check List

• [ ] All 5 bug programs located and fixed
• [ ] For every fix you can explain "why the original code misbehaves under a particular interleaving"
• [ ] You can distinguish the defect types TSan and helgrind are each best at detecting
• [ ] Bug 1: you can explain why a non-atomic ++counter is UB under multiple threads
• [ ] Bug 2: you can explain how waiting on a predicate solves both spurious wakeups and lost wakeups
• [ ] Bug 3: you can draw the resource allocation graph of the deadlock (circular wait)
• [ ] Bug 4: you can explain the lifetime risk of detach plus capture by reference
• [ ] Bug 5: you can back up the cache miss change with perf numbers, not just "it got faster after padding"
• [ ] All fixed code runs without a single TSan report