Lab 2.5: Concurrency Debugging Lab

Objectives

In previous Labs, we have been practicing "how to write concurrent code correctly". This Lab flips the script—we are facing code that is already written, but it contains bugs. Your task is not to implement from scratch, but to use tools and methodologies to locate issues, understand root causes, fix them, and perform regression verification.

This Lab provides five "intentionally broken concurrent programs," each containing a known type of concurrency defect—data race, lost wakeup, deadlock, use-after-free, and false sharing performance pitfalls. You need to go through the complete debugging loop of "locate → hypothesize → verify → fix" without looking at the answers first. The code structure reuses the patterns you are familiar with from Labs 0–2 (thread pools, queues, atomic counters), so the learning curve is low, and you can focus entirely on the debugging process itself.

Prerequisites

Before starting, ensure you have read the following chapters:

• ch08-01: Concurrent program debugging techniques — TSan, helgrind, custom logging
• ch08-02: Concurrent performance testing and benchmarking — perf, performance analysis methods
• Lab 0–2: Understand the basic structure of std::jthread, std::atomic, and std::latch

Environment Setup

The core of this Lab is toolchain configuration. You need the following tools:

• TSan: Compile with GCC/Clang using -fsanitize=thread
• helgrind: Part of Valgrind, valgrind --tool=helgrind
• perf: Linux performance analysis tool, perf stat and perf record

Install Valgrind (if not already installed):

sudo apt install valgrind

Debugging Methodology

Before diving into each buggy program, let's establish a unified debugging workflow. Every time you encounter a concurrency issue, proceed with the following steps:

Step 1: Confirm if it is really a concurrency issue. Run the same logic with a single thread. If the result is correct, then it is indeed a defect introduced by concurrency.

Step 2: Narrow down the reproduction scope. Reduce the number of threads, data volume, and run iterations to find the minimal reproduction path. Smaller reproduction code is easier to locate.

Step 3: Select the right tool. Use TSan for data races, helgrind or std::scoped_lock for deadlocks, and perf for performance anomalies.

Step 4: Interpret the reports. What do "previous write" and "current read" in a TSan report mean? How do you read the lock order graph in a helgrind report?

Step 5: Regression after fixing. Don't just "run it once." Run the full test suite under TSan to ensure the problem is completely eliminated.

Bug 1: Data Race on Shared Counter

Symptoms

Multiple threads modify the same int counter without locks, and the final result does not equal the expected value. Running it multiple times yields different results each time.

Defective Code

Expand (21 lines)Collapse

#include <atomic>
#include <iostream>
#include <thread>
#include <vector>

int main() {
    int counter = 0; // Data race: non-atomic shared variable
    std::vector<std::jthread> threads;

    for (int i = 0; i < 8; ++i) {
        threads.emplace_back([&counter] {
            for (int j = 0; j < 1000000; ++j) {
                counter++; // Unsafe concurrent write
            }
        });
    }

    // threads join automatically here (RAII)
    std::cout << "Final counter: " << counter << std::endl;
    return 0;
}

Debugging Tasks

1. Run the program and record the difference between the actual output and the expected value.
2. Run with TSan and interpret the data race location indicated in the report.
3. Fix the code (replace int with std::atomic<int>).
4. Choose the appropriate memory order (hint: std::memory_order_relaxed is sufficient for pure counting).
5. Perform regression verification with TSan.

Verification

After the fix, running 10 times consecutively should yield results equal to 8 * 1000000 (8,000,000). TSan should report no errors.

Bug 2: Lost Wakeup

Symptoms

The producer calls notify_one() before the consumer enters wait(), causing the consumer to block permanently. The program hangs.

Defective Code

Expand (30 lines)Collapse

#include <condition_variable>
#include <iostream>
#include <mutex>
#include <thread>

std::mutex mtx;
std::condition_variable cv;
bool ready = false;

void producer() {
    std::lock_guard<std::mutex> lock(mtx);
    ready = true;
    cv.notify_one(); // Notification happens here
    // If consumer hasn't started waiting yet, the signal is lost
}

void consumer() {
    std::unique_lock<std::mutex> lock(mtx);
    // Lost wakeup: if notify_one() was called before this line,
    // this wait will block forever.
    cv.wait(lock, [] { return ready; });
    std::cout << "Processing data..." << std::endl;
}

int main() {
    std::jthread prod(producer);
    // Simulate timing: sleep to ensure producer runs first
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
    std::jthread cons(consumer);
}

Debugging Tasks

1. Run the program multiple times and observe if it hangs every time (depends on thread scheduling).
2. Use timeout logging to assist diagnosis: add a wait_for timeout to cv.wait(), and print a log if it times out.
3. Fix the code: Change cv.wait(lock) to cv.wait(lock, predicate).
4. Explain why predicate waiting can solve both spurious wakeups and lost wakeups.

Verification

After the fix, the program should exit normally within 1 second. Try different thread start orders to ensure it runs correctly.

Bug 3: Deadlock from Lock Ordering

Symptoms

Two threads acquire two mutexes in reverse order. Under specific scheduling, a deadlock occurs. The program hangs.

Defective Code

Expand (26 lines)Collapse

#include <iostream>
#include <mutex>
#include <thread>

std::mutex mtx_a;
std::mutex mtx_b;

void thread1() {
    std::lock_guard<std::mutex> lock_a(mtx_a);
    std::this_thread::sleep_for(std::chrono::milliseconds(10));
    std::lock_guard<std::mutex> lock_b(mtx_b); // Potential deadlock point
    std::cout << "Thread 1 acquired both locks" << std::endl;
}

void thread2() {
    std::lock_guard<std::mutex> lock_b(mtx_b); // Locks B first
    std::this_thread::sleep_for(std::chrono::milliseconds(10));
    std::lock_guard<std::mutex> lock_a(mtx_a); // Potential deadlock point
    std::cout << "Thread 2 acquired both locks" << std::endl;
}

int main() {
    std::jthread t1(thread1);
    std::jthread t2(thread2);
    // Deadlock: t1 holds A, wants B; t2 holds B, wants A
}

Debugging Tasks

1. Run the program multiple times and observe if it occasionally hangs (deadlock requires a specific scheduling order).
2. Run with helgrind: valgrind --tool=helgrind ./your_program, and interpret the lock order conflict report.
3. Use std::scoped_lock to acquire both locks simultaneously, eliminating the lock ordering issue.
4. Alternatively, unify the locking order for both threads (both A then B).

Verification

After the fix, it should not hang even after running 100 times consecutively. helgrind should no longer report lock order conflicts.

Bug 4: Use-After-Free in Detached Thread

Symptoms

A detached thread continues to access a local variable that has been destroyed. The program may crash, output garbage values, or behave normally—behavior depends entirely on timing.

Defective Code

#include <iostream>
#include <string>
#include <thread>

void task() {
    std::string message = "Hello from background";
    std::jthread t([&message] { // Capture by reference
        // Risk: 'message' is destroyed when task() returns
        // but this thread might still be running.
        std::cout << message << std::endl;
    });
    t.detach(); // Detach the thread
} // 'message' is destroyed here, but the detached thread might still access it

int main() {
    task();
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
    return 0;
}

Debugging Tasks

1. Run the program multiple times—sometimes output is normal, sometimes garbled, sometimes segmentation fault.
2. Run with TSan and view the use-after-free report (TSan can detect accesses to freed memory).
3. Fix the code: Use value capture instead of reference capture [message], so the thread owns its own copy.
4. Alternatively, replace detach with join to ensure the thread completes before the variable is destroyed.

Verification

After the fix, running 50 times consecutively should output "Hello from background" normally. TSan should report no errors.

Bug 5: False Sharing Performance Trap

Symptoms

Two threads modify atomic variables in adjacent memory locations. Performance is far below expectations. Functionally correct, but throughput is even lower than the single-threaded version.

Defective Code

Expand (36 lines)Collapse

#include <atomic>
#include <chrono>
#include <iostream>
#include <thread>
#include <vector>

struct Counter {
    std::atomic<int> a;
    std::atomic<int> b; // False sharing: 'a' and 'b' share the same cache line
};

Counter counter;

void worker_a() {
    for (int i = 0; i < 10000000; ++i) {
        counter.a.fetch_add(1, std::memory_order_relaxed);
    }
}

void worker_b() {
    for (int i = 0; i < 10000000; ++i) {
        counter.b.fetch_add(1, std::memory_order_relaxed);
    }
}

int main() {
    auto start = std::chrono::high_resolution_clock::now();

    std::jthread t1(worker_a);
    std::jthread t2(worker_b);

    auto end = std::chrono::high_resolution_clock::now();
    std::cout << "Time elapsed: "
              << std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count()
              << "ms" << std::endl;
}

Debugging Tasks

1. Run this version first and record the time taken.
2. Fix: Add cache line padding between the two atomics (alignas(64) or manual padding).
3. Use perf stat to observe the change in cache miss counts before and after the fix.
4. Compare the time taken before and after the fix and calculate the speedup.

The fixed structure should look like:

struct Counter {
    alignas(64) std::atomic<int> a;
    char padding[64 - sizeof(std::atomic<int>)]; // Manual padding
    std::atomic<int> b;
};

Verification

The time taken after the fix should be 2-5 times faster than before (depending on CPU architecture). Observing the cache-misses metric with perf stat should show a significant decrease.

Self-Check List

• [ ] All 5 bug programs located and fixed.
• [ ] For every fix, you can explain "why the original code fails under specific timing".
• [ ] Can distinguish the defect types that TSan and helgrind are good at detecting.
• [ ] Bug 1: Can explain why non-atomic int is UB in multithreading.
• [ ] Bug 2: Can explain how predicate waiting solves both spurious and lost wakeups.
• [ ] Bug 3: Can draw the resource allocation graph for the deadlock (circular wait).
• [ ] Bug 4: Can explain the lifetime risk of detach + reference capture.
• [ ] Bug 5: Can use perf data to demonstrate cache miss changes, not just "it got faster after padding".
• [ ] All fixed code passes TSan without reports.