Most benchmarks compare by subtraction. That is the bug.

The benchmark that changes its mind.

You have a change you believe is faster. You run the benchmark: old path for a few seconds, new path for a few seconds, divide. Eight percent better. You merge it. A week later, same revision, same benchmark, and now the new path is slower.

Everyone who has optimized anything has had this week. The usual response is to reach for a better statistical test, and the usual result is a more confident wrong answer. The defect is not in the analysis. It is one line above the analysis, in a step so ordinary that almost nobody writes it down:

t_a = time(A)      # seconds 0 through 3
t_b = time(B)      # seconds 3 through 6
speedup = t_a / t_b

That division is only meaningful if the machine at second 0 and the machine at second 3 are the same machine. On a laptop that decided to index your mail, or a CI runner sharing a socket with somebody else's link step, they are not. The ratio you computed contains the difference between your two implementations and the difference between two moments, and nothing in the output tells you which one you are looking at.

This is not an exotic failure. It is the default failure of the entire category. Criterion.rs compares your run against a baseline saved to disk, possibly days ago. Go's benchstat compares two files. Google Benchmark's compare.py compares two JSON documents. JMH forks a separate JVM per benchmark and compares the results afterwards. Every one of those is a subtraction across time, and every one inherits the same hole.

flyology_bench starts from the position that a comparison is not an arithmetic operation you perform on two measurements. It is a measurement, with its own design, its own schedule, and its own statistics — and once you build it that way, six other things become possible that a subtractive harness cannot offer at all.

Stop subtracting. Start pairing.

In Flyology_Bench you do not measure twice and divide. You instantiate Compare over both operations, and the harness collects the comparison as a single measurement.

procedure Compare_Paths is new Flyology_Bench.Compare
  (Reference_Operation => Existing,
   Contender_Operation => Rewrite);

Result : Flyology_Bench.Comparison;

Compare_Paths (Result => Result);
Flyology_Bench.Reporters.Put_Comparison_Console
  ("existing", "rewrite", Result);

The reference and the contender now run adjacently, inside every sample pair. Fifty pairs, alternating, over the same second of wall-clock time. Drift that moves the machine moves both halves of a pair together, so it cancels inside the ratio instead of accumulating into it. Every one of the fifty ratios is local in time, and the estimate is built from ratios rather than from two distant averages.

Adjacency alone would still be biased, because whichever side runs first pays for the other side's cache and frequency state. So the harness deterministically shuffles which side leads and holds the two order counts equal, or within one sample when the count is odd. It then reports the residual: Order_Effect_Percent is the gap between the reference-first and contender-first ratio groups. If your two implementations interact through the cache in a way that balancing cannot remove, that number is where it shows up, rather than quietly living inside your speedup.

The second default is the one people are surprised by. Each side calibrates its own iteration count toward the same timed slice, so a contender that is twice as fast performs twice as many logical operations rather than finishing early and collecting half the data. An equal-iteration schedule does the opposite: it systematically gives the faster implementation less measurement time, which is precisely backwards. Shared_Iterations is available when one logical operation count is genuinely the point.

Then the pairing survives into the statistics, which is the step most tools skip even when they interleave. Confidence bounds come from deterministic circular-block resampling of the paired log-ratios, so both pair membership and short-range sample order are preserved through the bootstrap. Lag-one correlation of the paired samples is reported next to the interval. You are not asked to trust that the schedule was fair; you are handed the diagnostics that would show it was not.

Finally, the harness commits to a verdict, and refuses to when it should:

Google Benchmark deserves credit as the closest thing to prior art: --benchmark_enable_random_interleaving interleaves repetitions across benchmarks and its documentation reports about 40 percent lower run-to-run variance. It is opt-in, it randomizes positions rather than balancing them, and its comparison tooling still treats the two sides as independent samples afterwards. The interleaving improves the inputs; it never reaches the statistic. Here, pairing is the default and the pairing is what the interval is computed from.

Every axis rides the same schedule.

Now the part that a subtractive harness structurally cannot give you. Once a comparison is one scheduled measurement rather than two, anything you sample around those same batches inherits the same pairing — and therefore gets its own paired verdict.

use type Flyology_Bench.Metric_Set;

Config.Metrics :=
  Flyology_Bench.Process_Resource_Metrics
  or Flyology_Bench.Linux_Hardware_Metrics;

A Metric_Set selects the axes retained around those same timed batches. The portable set covers process and thread CPU time, resident memory and its change, minor and major faults, voluntary and involuntary context switches, storage bytes, and filesystem operations. Linux_Hardware_Metrics adds CPU cycles, retired instructions, instructions per cycle, cache misses, branches, and branch misses through perf_event_open. Wall time remains the calibration and budget clock throughout.

The result is that the annoying outcome stops being annoying. "Same wall time" is normally where a benchmarking session dies. Here it is the start of the diagnosis: same wall time, instructions per cycle up eleven percent, cache misses down by a third, involuntary context switches unchanged — you did not fail, you became memory-bound somewhere else. Each of those statements arrives with a paired confidence interval from the same fifty pairs, not from a second run you would then have to defend.

That recording is the maintained example running end to end, captured on a pty by scripts/record-bench-cast.sh so the in-place progress line behaves as it does for a person. Watch the axis tables scroll past: on this Apple Silicon host every hardware row reads unavailable: unsupported platform rather than zero, which is the behavior described below and the reason the recording was worth keeping rather than staging.

The arithmetic is careful in ways that matter. Positive-only axes are compared as relative ratios; signed or zero-containing axes use paired absolute differences, because a ratio through zero is nonsense. Every delta is divided by the logical operation count, except absolute resident memory and the dimensionless IPC value. Axes with no general optimization direction get a Diagnostic verdict rather than a fabricated better-or-worse.

Hardware counters themselves are well-trodden ground and we will not pretend otherwise: Google Benchmark reads them through libpfm, nanobench prints them per operation, and JMH's -prof perfnorm normalizes them to iterations. What none of them will do is hand you a paired, order-balanced, bootstrapped verdict on cache misses collected inside the same schedule that produced the timing verdict. That is what falls out of treating the comparison as the primitive.

Sixteen implementations, one schedule.

The pairing generalizes. Compare_Many takes an enumeration of two to sixteen implementations, calibrates each toward an equal share of the timed budget, and then places every case once in each execution position over shuffled cyclic rounds.

type Candidate is (Existing, Rewrite, SIMD, Tasked);

procedure Batch
  (Which : Candidate; Iterations : Flyology_Bench.Iteration_Count);

procedure Compare_All is new Flyology_Bench.Compare_Many
  (Case_Id => Candidate, Batch => Batch);
procedure Put_All is new
  Flyology_Bench.Reporters.Put_Multi_Comparison_Console (Candidate);

Result : Flyology_Bench.Multi_Comparison;

Compare_All (Result => Result);
Put_All (Result, Show_Individual_Details => True);

Position balancing is the point. Run four implementations as four consecutive blocks and the last one competes against a warmer, hotter, more thermally throttled machine than the first; the ordering of your results is then partly the ordering of your source file. Rotating every case through every position removes that, and Sequential_Cases remains available when you want per-case telemetry blocks and accept the exposure.

Look at row three. Its median is lower than the reference, and the harness still refuses to call it a win, because the interval straddles parity. A tool that ranked by median would have handed you a 2.3 percent improvement to put in a pull request description. This one tells you that you measured nothing, which is the more valuable sentence.

The machine does not stay quiet, so we watch it.

Pairing removes drift between the two sides. It does not remove a compile job that lands in the middle of your run and inflates both sides at once. Most harnesses respond to this by printing a warning at startup and producing a number anyway. Two go further, and it is worth naming them: pyperf will actually quiet the machine for you with pyperf system tune, given root and a box you may mutate; Google Benchmark warns when CPU scaling is on. Neither watches while it measures.

The first line here is a preflight. CPU_Quiescence waits for both the host-wide average and the busiest single logical CPU to stay under their limits for a continuous interval before clock characterization and warmup begin, and raises CPU_Quiescence_Timeout rather than collecting under load it already knows about. Two limits, not one, because a single saturated core hides easily behind fifteen idle ones.

The gate has a hole, and it is the interesting one: it describes the past. So the harness keeps looking.

Config.Interference :=
  (Enabled                     => True,
   Response                    => Flyology_Bench.Retake,
   Maximum_Foreign_CPU_Percent => 10.0,
   Window                      => 0.050,
   Maximum_Retakes             => 25,
   Settle_Time                 => 0.250,
   Maximum_Pause_Time          => 30.0,
   Rewarm_Time                 => 0.050);

The obvious implementation does not work, which is probably why nobody ships it. Host utilization cannot be compared against a limit during collection the way the preflight does it, because by then your benchmark thread is saturating a core and every window trips. Foreign load has to be defined as host busy time minus this process's own CPU time, so the load your benchmark creates is correctly not counted against it. Measured against top on a deliberately loaded machine, the two estimates agree to within 0.1 percent.

With a workable definition, three responses form a ladder. Observe keeps every sample and records what it saw. Retake discards the contaminated window and collects it again. Pause waits for the host to settle, re-warms, and resumes. The re-warm is not politeness: a resumed run has cold caches, cold predictors, and a different frequency state, so without it your first sample after a pause is exactly the outlier the pause existed to prevent.

Windows cover whole collection units — one sample, one comparison pair, or one balanced multi-way round — so a response can never split the two halves of a pair or land mid-round, where it would spread interference unevenly across the cases the round exists to compare fairly. Discarding a window rolls back what that window changed, including the position tallies and telemetry sums, because otherwise a run with eighteen retained samples and twenty-four retakes would report an elapsed time covering forty-two samples, most of them thrown away. Budgets degrade rather than abort: an exhausted retake budget continues under Observe and says so.

What you get out of this is auditable CI. Environment carries the contaminated sample count, retakes, pauses, and the mean and peak foreign share into the console card, the CSV, and the JSON. A merge-blocking benchmark can require that its result had zero contaminated windows. A flaky one can be diagnosed as a statement about your runner rather than about your code. A run that had to repair itself says so, because otherwise a heavily repaired result reads as a quieter machine than the one it actually ran on.

A convention for machines that host more than one job.

Watching the host creates a new failure, and it is a good one. Two harnesses that both pause on interference are symmetric controllers observing each other: each pauses because the other is loading the machine, both then observe quiet, both resume together, and both immediately pause again. Solving that required a protocol, and there was not one to adopt.

Host_Lock claims host CPU capacity for the duration of a run, taken before the preflight gate — waiting for a quiet host and only then blocking on a claim would leave the quiet verdict stale by the time collection started. The convention is deliberately named for the resource rather than for benchmarking, so the tools that create load — a load generator, a soak test, a profiler — take the same claim as the tools that need its absence. The file naming, protocol, and content grammar are published so your own tooling can participate.

Conflict detection is flock and nothing else, which is what makes it safe to adopt. Liveness is the lock itself, so a job the CI system kills releases with no cleanup path, no stale registry, and no pid-reuse hazard. A machine-wide claim takes an exclusive lock on the claim file. A per-CPU claim takes a shared lock there plus an exclusive lock on one file per CPU, acquired in ascending order, because a total order is what stops two callers with overlapping sets from deadlocking on each other's partial claims. Disjoint core-scoped claims therefore proceed concurrently while a machine-wide claim excludes them all, with no arbiter anywhere.

Even the path is a decision rather than a convenience. It is a single deterministic default, not a writability-probing fallback chain, because a chain reintroduces exactly the failure it was meant to prevent: /run/lock is mode 1777 on Debian and root-owned 0755 on Red Hat derivatives, so a root process and an unprivileged process on one machine would probe differently, land on different files, and each conclude it holds the machine. File content is diagnostic and parseable but never consulted to decide a conflict, and readers open it without locking, since a shared lock would block behind the very holder they are trying to name.

And the claim knows what it is not. systemd PrivateTmp= bind-mounts a per-service directory over /tmp, so two services share every CPU and cannot see each other's claim files; that case is detected from /proc/self/mountinfo and reported as namespace-scoped. A path can be proven private but never proven machine-wide. Require_Machine_Scope exists so a merge-blocking job can refuse to run rather than proceed unserialized.

Every number carries its own provenance.

Collecting a value and being entitled to attribute it are different questions, and almost every telemetry-carrying benchmark conflates them. A process-wide resident-memory figure can be collected perfectly and still not be your operation's memory. So each axis here carries four independent facts: a status, a scope, an attribution quality, and an optimization direction.

That structure buys specific refusals. Thread CPU is accepted only when both boundaries execute on the same native thread; a migrated span drops its thread and counter values rather than attaching them to the destination thread. Process-scoped axes are labeled as covering the whole process, so nobody quotes them as per-operation. And when a counter is unavailable, the harness never substitutes zero — because you would read that zero as "no cache misses" and nothing downstream would ever correct you.

The failure classification is unusually specific for the same reason. Metric_Status distinguishes an unsupported platform, a permission denial, an event the host's monitoring unit does not implement, unavailable counter resources, and a failed probe. perf_event_open reports EINVAL both for a generic event the host cannot map and for an attribute combination the kernel rejects, so that case is re-probed with only the permission-relevant attributes before being classified — the difference between "your kernel will not let you" and "this chip has no such counter" is the difference between a fixable CI config and a wasted afternoon.

The same discipline governs the words in the report, which is where microbenchmarking most often misleads people who did nothing wrong:

-- integer_mix -----------------------------------------
   latency    | median 1.498 ns/op  mean 1.633 ns
   tails      | p95 2.225 ns  p99 2.784 ns  range 1.454 ns .. 2.839 ns
   sampling   | 16777216 iter/sample x 55 samples  median batch 25.13 ms
   quality    | CV 18.60%  outliers 7  lag-1 correlation 0.47
   clock      | mach_absolute_time
   resolution | nominal 42.000 ns  observed 41.000 ns  median read 41.000 ns  floor 0.003 ps/op
   cpu        | ▇▇▇▇▇▇▇▇▁▇▇▇▇▇▇▇▇▇▇▇▇▇▁▇▇▇▇▇▇▇▇▇▇▇▇▇▁▇▇▇▇▇▇▇▇▆▃▄
              | average 96.8%  (1.0 cores)    peak 100.0%  (1.0 cores)
   memory     | ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
              | RSS change across own batches +0.0 KiB    largest batch +0.0 KiB
   elapsed    | 00:00:01.5 timed samples (hh:mm:ss)

Note what that card refuses to hide. A CV of 18.6 percent and a lag-one correlation of 0.47 are printed at the same size as the median. The clock characterization sits directly beneath the tails, so a reader can see that the estimate rests on a 42 ns timer amortized across sixteen million iterations. Nothing here is buried in a verbose flag.

Machine-readable output preserves all of it. Set FLYOLOGY_BENCH_OUTPUT=csv or json for clean stdout with no progress or ANSI sequences; the long-form reporters emit one row per axis including availability status and failure reason, and the crate's test suite checks that every CSV row carries exactly the columns its header declares and that available agrees with status.

The same tool follows you into production.

Everything so far assumes the harness is the caller. For a server that assumption is false: the request arrives when it arrives, and the code you want to measure is already running. This is normally where you abandon your benchmarking library and start assembling timers, a histogram crate, and a metrics exporter — three new dependencies with three different opinions about what a percentile is.

Flyology_Bench.Recording inverts the control and keeps the axes, the status vocabulary, and the reporters.

Recorder : Recording.Recorder
  (Maximum_Benchmarks => 8, Retained_Samples => 10_000);
Request : Recording.Benchmark;

Recording.Register (Recorder, "request", Request);
Recording.Start (Recorder);

declare
   Sample : Recording.Span;
begin
   Recording.Begin_Sample (Recorder, Request, Sample);
   Handle_Request;
   Recording.Finish (Sample, Recording.Success);
end;

Registration and bounded-store allocation happen before Start, so the boundary path performs no Ada heap allocation. Finish reads the ending timestamp before entering the protected sample store, which keeps retention and analysis out of the value you are measuring. Retention is bounded and the policy is named — Reservoir for a long run, with First_N and Latest_N available — and omitted samples are counted rather than silently dropped.

Because the recorder times individual spans rather than batches, this is also where real per-invocation percentiles live. Each retained row keeps its observation number and outcome next to the status and value of every requested axis, so the raw CSV and the JSON samples array preserve the relationship between latency, CPU, memory, and outcome even when a thread migration makes one axis unavailable for that row alone. Filtering to successful spans keeps every other column aligned, which is the query you always end up needing at 2am.

On Linux each recorder session owns PMU groups that stay enabled per native worker, and worker exit closes that worker's groups so later pthread reuse cannot inherit stale descriptors. Concurrent recorders use separate sessions. This is production-shaped machinery, not a demo.

The live display is the part a screenshot cannot carry: fixed rows, refreshed in place, with completed and active span counts, errors, and rolling median and p95 moving as the four service workers run. A still frame of the same display looks like this:

fly recorder  elapsed 01.31 s      cpu  38.4% / 0.4 cores   rss 31.2 MiB
benchmark                    done  live  errors   median      p95
quick request                  66     0       2   42.0 ns     250.0 ns
cpu-heavy request              68     1       1    3.21 ms      4.08 ms
memory-burst request           70     0       2  140.3 us     358.2 us
wait-bound request             64     1       2   19.01 ms     64.03 ms

And the recorder refuses to borrow the runner's statistics, which is what keeps the whole thing coherent. Recorded percentiles describe individual spans, and the output marks that contract as sample_semantics=individual_span. Compare_Independent resamples independently and is marked comparison_design=independent, because two separately observed distributions are not a paired design, and reusing the paired bootstrap on them would hand you a category error with a confidence interval attached.

What it adds up to.

Take the pieces together and the shape is clear. One instantiation gives you a paired, order-balanced, equal-time comparison whose confidence interval is computed from the pairing itself; the same fifty pairs also yield paired verdicts on cycles, instructions per cycle, cache misses, faults, context switches, resident memory, and storage traffic, each carrying its own status, scope, and attribution; the host is gated before the run and watched throughout it, with contaminated windows repaired or annotated and never silently corrected; runs coordinate with other tools through a published claim convention; and when the workload stops being callable, the same axes and the same reporters follow it into a live service.

We did not invent most of the individual ideas and the debts are deliberate. Do_Not_Optimize and Clobber_Memory carry Google Benchmark's names on purpose, so anyone arriving from C++ knows what they do before reading a line of documentation. The four-bucket Tukey classification and the clock characterization pass came through nonius to Catch2 and to both criterions. The adaptive batch is Go's b.N with a different calibration target. Paired designs are ordinary statistics, and the literature has argued for randomized interleaving for years.

What we could not find anywhere, and the reason this crate exists, is a general-purpose harness that does all three of these at once:

1. Makes paired, order-balanced comparison the default primitive and carries the pairing through to the confidence interval, rather than comparing two runs collected at different times.
2. Retains resource and hardware axes on that same sample schedule, each with its own status, scope, and attribution, and never substitutes zero for an absence.
3. Watches the host during collection and repairs or annotates a contaminated window while refusing to adjust any reported statistic.

If you know of one, we would rather use it than maintain this. Until then, this is the harness we would reach for on a machine we do not control, chasing a difference small enough that the machine could swallow it, for a result that has to survive review.

alr index --reset-community
alr index --add=git+https://github.com/flyology-ada/alire-index.git \
  --name=flyology --before=community
alr with flyology_bench

The benchmarking guide walks through a first measurement, and the generated API reference documents every contract named here.

What this record does not establish.

This is a design record, not a measurement. It reports no throughput result and supports no comparison between Ada and any other language. flyology_bench is version 0.1.1-dev, distributed through a development index, and carries no stability promise.

Platform coverage is uneven. Hardware counters and strict thread placement are Linux features; Darwin's affinity interface is an advisory grouping hint rather than a CPU name, and Apple Silicon implements no thread affinity at all and rejects every request, so Require_Strict turns that into a refusal instead of a silent downgrade. A Linux guest with no exposed monitoring unit reports the specific unavailable status rather than a pass.

The host model is a CPU model. It detects competing CPU work. It does not detect storage traffic, thermal state, frequency transitions, interrupt load, or contention for shared cache and memory bandwidth — and that last one perturbs your measurement without moving any CPU busy counter. A host claim coordinates only with processes that reach the same file. Enabling several probe families can perturb another family at the boundary, so when a small resource difference matters, repeat the comparison with only that family selected and confirm the conclusion survives.

Some things are deliberately out of scope and other tools do them better: allocation counts per operation belong to Divan and Go's -benchmem; near-deterministic instruction counts in noisy CI belong to iai-callgrind; tracking a benchmark across a commit range belongs to airspeed velocity, where we have only fingerprinted Flyology_Bench.Baselines; JVM benchmarking belongs to JMH. This is not a profiler either: it tells you a difference exists and how confident it is, not where it came from.

What it will not do is turn a busy laptop into a quiet server. What it does instead is put the noise in the report rather than in the number — which is the difference between a result you can argue with and one you can only believe.