Benchmark methodology

This is the anchor chapter of the whole volume. ch00 laid down the two iron rules — "correct first, then fast" and "measure first, then optimize" — but behind that little phrase "measure first" hides an entire discipline: performance is not a boolean, it is a distribution; a one-off hand-rolled measurement is almost meaningless; and the microbenchmark, the handiest tool of them all, is also precisely the one most likely to lie to you.

This chapter tears "measuring accurately" down to the studs: first see through the three kinds of deception a microbenchmark can pull, then learn how to write one that doesn't lie (the semantics of DoNotOptimize, parameter sweeps, repetition and aggregation), then work through a 16-item environment-readiness checklist that shuts noise sources down one by one, then use statistics to turn the distribution into a trustworthy conclusion, and finally cover how to move measurement into production and CI to keep standing guard over performance. Every performance article after this one opens by citing back to this chapter's rules — the way vol5 threads TSan through concurrency correctness.

If you only read a few pieces from this volume, this chapter should account for most of them.

In this chapter

STARTFINISHWhy microbenchmarks lie→STARTFINISHHow to write a credible microbenchmark→STARTFINISHMeasurement pitfalls and environment readiness: a 16-item checklist→STARTFINISHStatistics and reporting: turning a distribution into a conclusion→STARTFINISHProduction measurement and CI performance regression detection→STARTFINISHBenchmark methodology reference card→