SIMD and vectorization: auto-vectorization conditions, intrinsics, and CPU dispatch

One instruction, multiple data

SIMD (Single Instruction, Multiple Data) is the contemporary CPU's main lever for more compute throughput: one instruction processes multiple data elements at once. The wider the register, the more data per instruction:

• SSE: 128-bit registers, 4 floats at a time (or 2 doubles).
• AVX/AVX2: 256-bit, 8 floats at a time.
• AVX-512: 512-bit, 16 floats at a time.

In theory, converting a scalar loop to AVX2 multiplies throughput directly by ×8 (8 floats' worth of arithmetic per instruction). We measure a dot product to see the real number:

===== SIMD dot product (4M floats = 16MB, average of 30 runs) =====
  scalar dot_scalar               958.2 us
  hand-written AVX2 dot_avx2       48.1 us
  AVX2/scalar = 19.9x

Close to 20x. This number is higher than the theoretical 8x because the hand-written version, on top of the SIMD width, also uses FMA (one instruction does the work of a multiply and an add) and multiple accumulators (breaking the dependency chain, per 04-02) — all three stacked. First, correct one piece of intuition: the scalar dot_scalar at -O3 -mavx2 is already auto-vectorized (g++ -O3 -mavx2 -fopt-info-vec-optimized reports loop vectorized using 32 byte vectors), it's not "not vectorized." So why is it still 20x slower? Because its accumulation forms a dependency chain (the assembly below shows it's a scalar horizontal accumulation chain), and the hand-written dot_avx2 has 4 parallel accumulators. This 20x is mostly an ILP gap of "multiple accumulators vs single accumulator," not "vectorized vs not." That's the point this article is built to make.

Auto-vectorization: conditions + a common misconception about FP reductions

The compiler at -O2/-O3 -ftree-vectorize tries to auto-convert loops to SIMD. Preconditions:

1. Iteration count is estimable (the compiler needs a rough idea of how many to schedule SIMD).
2. No pointer aliasing (the compiler has to assume the memory accessed by two loop iterations doesn't overlap; __restrict helps, see ch04-04).
3. No function calls (or the calls can be inlined away).
4. Simple branches (no complex control flow).

The fifth condition, "FP reduction," deserves an update from the old wisdom. The core of a dot product is acc += a[i] * b[i], an FP reduction. Vectorizing it means splitting the accumulator into parallel lanes, which changes the association order of floating-point addition, and FP addition is not associative ((a+b)+c ≠ a+(b+c), different mantissa rounding errors). Old textbooks often say "default -O3 strictly respects FP semantics, doesn't dare change the association order, so it doesn't vectorize FP reductions" — that claim is outdated on modern GCC: GCC 10 onward uses the ordered (order-preserving) reduction mode to vectorize FP reductions, preserving the observable accumulation order (bit-exact result) and only using SIMD internally for the order-preserving partial sums. Verify it yourself: g++ -O3 -mavx2 -fopt-info-vec-optimized on this article's dot_scalar reports loop vectorized using 32 byte vectors.

So why is the scalar dot product still nearly 20x slower than hand-written dot_avx2? Look at the assembly the ordered mode actually produces: at -O3 -mavx2 the multiply in dot_scalar is vectorized with vmulps (8-way SIMD), but the order-preserving accumulation is a chain of scalar vaddss (lane-by-lane horizontal adds into the same xmm0) — a scalar add dependency chain. That's the cost of ordered mode: it vectorized the multiply, but couldn't parallelize the accumulation (parallel accumulation would change the FP association order). The hand-written dot_avx2's 4 vector accumulators are what break that limit and let the 4 FMAs actually run in parallel. So this 20x is mostly the gap of "4 parallel accumulators vs scalar sequential accumulation," not "vectorized vs not." If you want the compiler to split the lanes itself, you have to give it -ffast-math (or the narrower -fassociative-math) to loosen FP semantics. With -ffast-math, GCC does rearrange dot_scalar into 4 vector accumulators (assembly shows vfmadd231ps × 4 + unroll 32, same structure as hand-written dot_avx2); but on this WSL2 machine the measured time barely changes, which means the 20x gap on this simple loop isn't only about accumulator count — dot_scalar gets inlined into the REP outer loop, while dot_avx2 as an independent function call sits in a different measurement structure. Integer reductions don't have the associativity problem, default to multi-lane, and auto-vectorize more aggressively.

To know whether your loop vectorized and why it didn't get the full speedup, use GCC's diagnostics: -fopt-info-vec-optimized (what got vectorized) and -fopt-info-vec-missed (why not) — the fastest way to read the compiler's mind.

Hand-written intrinsics: AVX2 in practice

When auto-vectorization can't get there (non-standard access patterns, specific instructions, or you just want to squeeze hard), hand-write intrinsics. intrinsics are compiler-builtin C functions that map to single SIMD instructions, header <immintrin.h>. The core of an AVX2 dot product:

#include <immintrin.h>
float dot_avx2(const float* a, const float* b) {
    __m256 v0 = _mm256_setzero_ps(), v1 = _mm256_setzero_ps(),
           v2 = _mm256_setzero_ps(), v3 = _mm256_setzero_ps();
    for (int i = 0; i < N; i += 32) {           // 4 vectors × 8 floats = 32
        v0 = _mm256_fmadd_ps(_mm256_loadu_ps(a+i),    _mm256_loadu_ps(b+i),    v0);
        v1 = _mm256_fmadd_ps(_mm256_loadu_ps(a+i+8),  _mm256_loadu_ps(b+i+8),  v1);
        v2 = _mm256_fmadd_ps(_mm256_loadu_ps(a+i+16), _mm256_loadu_ps(b+i+16), v2);
        v3 = _mm256_fmadd_ps(_mm256_loadu_ps(a+i+24), _mm256_loadu_ps(b+i+24), v3);
    }
    // horizontal merge of the 4 accumulators
    __m256 s = _mm256_add_ps(_mm256_add_ps(v0, v1), _mm256_add_ps(v2, v3));
    alignas(32) float tmp[8]; _mm256_store_ps(tmp, s);
    float acc = 0; for (int i = 0; i < 8; ++i) acc += tmp[i];
    return acc;
}

A few notes:

• __m256 is the 256-bit vector type, holding 8 floats.
• _mm256_loadu_ps: load 8 floats from memory (u = unaligned, allows unaligned; the aligned version _mm256_load_ps requires 32-byte alignment).
• _mm256_fmadd_ps(a, b, c): a*b + c, FMA does a multiply-add in one instruction (AVX2 + FMA only; compile with -mfma).
• 4 accumulators v0..v3: break the dependency chain (04-02) so 4 FMAs can fill the execution ports in parallel.
• Horizontal merge: a SIMD accumulator finally has to reduce to a scalar; this step has a lot of scalar ops, so don't open too many accumulators (8 doesn't beat 4).

This style squeezes the SIMD width + FMA + multiple accumulators dry and measures to 20x. The cost is poor portability (locked to AVX2+FMA; older CPUs can't run it) and poor readability. So the boundary for hand-written intrinsics is: don't hand-write what auto-vectorization can do; only hand-write when auto can't and you've confirmed it's a hotspot.

AVX-512: watch out for throttling

AVX-512 sounds even more aggressive (512-bit, 16 floats at a time), but there's a trap: on some Intel CPUs, running AVX-512 instructions triggers license-based frequency throttling — the CPU decides AVX-512's power/thermal requirements are high and actively drops the all-core frequency a step, and the throttle persists for a while even after the AVX-512 instructions end (waiting to "cool down"). The result: a small stretch of AVX-512 code can slow down the rest of the whole program, a net loss.

This trap has been improving across Intel generations (Ice Lake onward, lightweight AVX-512 doesn't downclock; Skylake-X was the worst), and AMD Zen 4 only introduced AVX-512 with a different policy. In practice, SSE/AVX2 + FMA is the safe sweet spot — the gain is already big and there's no throttle trap; AVX-512 is either for new CPUs confirmed not to throttle, or for the "lightweight" instruction subset, and must be benchmarked. The local 5800H is Zen 3 and only supports up to AVX2 (no AVX-512), so all SIMD examples in this volume are AVX2-level.

CPU dispatch: function multiversioning

Hand-written intrinsics lock the instruction set, hurting portability. The fix is CPU dispatch / function multiversioning: write multiple versions of the same function (SSE, AVX2, scalar) and pick at runtime by CPU capability. GCC's built-in __builtin_cpu_supports("avx2") queries capability; the more elegant __attribute__((target_clones("avx2","default"))) has the compiler auto-generate the versions plus the dispatch:

__attribute__((target_clones("avx2","default")))
float dot(const float* a, const float* b) { /* one piece of generic code, compiler compiles per target */ }

The compiler produces an AVX2 version and a default version, and on first call detects the CPU and picks (the IFUNC mechanism). One piece of source, each CPU runs its optimized version. The cost: a bigger binary (multi-version code) and a tiny dispatch overhead on first call.

Outlook: std::simd (C++26)

C++26 standardizes std::simd (in <experimental/simd>), providing a portable SIMD abstraction — you write "process with width-N SIMD" and the compiler maps it to each platform's concrete instructions. This offloads the portability problem of hand-written intrinsics to the standard library. vol6 only drops a one-line outlook here; depth waits for C++26 adoption.

Compressing this article: SIMD is theoretically ×4/×8/×16, and the measured hand-written AVX2 dot product gets ~20x (SIMD width + FMA + multiple accumulators stacked). The conditions for auto-vectorization are estimable iteration count, no aliasing, no complex branches; modern GCC (GCC 10+) vectorizes FP reductions in ordered mode by default (order-preserving, bit-exact result), but the single-accumulator dependency chain caps it, and getting the full SIMD needs multiple accumulators (hand-written, or -ffast-math so the compiler dares to split lanes) — it's not "turn on -O3/-ffast-math and you're done"; read the diagnostics with -fopt-info-vec. Hand-written intrinsics squeeze SIMD dry but hurt portability; only use them on hotspots where auto can't. AVX-512 has a throttle trap; SSE/AVX2+FMA is the safe sweet spot; you must benchmark. CPU dispatch (target_clones / __builtin_cpu_supports) solves portability; std::simd (C++26) is the future.

References

• Agner Fog, Optimizing assembly, §5 "Vector programming" + §13 "AVX-512"; local copy.
• GCC docs on -ftree-vectorize / -fopt-info-vec / target_clones / __builtin_cpu_supports.
• Intel Intrinsics Guide (intel.com/intrinsics-guide) — intrinsic lookup.
• Measured code for this article: code/volumn_codes/vol6-performance/ch04/simd.cpp.