Benchmark methodology

This is the volume's anchor chapter. ch00 established the two iron rules — "correct first, then fast" and "measure first, then optimize" — but behind "measure first" hides an entire discipline. Performance isn't a boolean, it's a distribution; a one-off hand-rolled measurement is almost meaningless; and the microbenchmark, the handiest tool in the box, is also the one most likely to lie to you.

This chapter tears "measuring accurately" down to the studs: first see through the three ways a microbenchmark deceives you, then learn how to write one that doesn't (DoNotOptimize semantics, parameter sweeps, repetition aggregation), then a 16-item environment-ready checklist to shut noise sources down one by one, then statistical methods to turn a distribution into a trustworthy conclusion, and finally how to move measurement into production and CI to stand guard over performance. Every performance article after this one opens by pointing back to the rules here — the way vol5 threads TSan through concurrency correctness.

If you only read a few chapters in this volume, this one should be most of them.

In this chapter

Why microbenchmarks lie to you→How to write a credible microbenchmark→Measurement pitfalls and environment readiness: a 16-item checklist→Statistics and reporting: turning a distribution into a conclusion→Production measurement and CI performance regression detection→Benchmark methodology reference card→