How we measure

Every figure we publish or deliver carries its context: what was measured, where, when, under which load and with which tool. This page describes the procedure we follow and the templates we use.

Three kinds of figures

Benchmark. A measurement in a controlled environment with synthetic load. Hardware, operating system, power profile, versions and load generator are declared. It compares implementations or configurations under the same conditions; it does not predict performance on other hardware or with real traffic.
Production. A measurement over real traffic, with the tool that recorded it, the period and the system. It sets the baseline and verifies the effect of a change.
Estimate. A calculation from explicit assumptions, for example message size and expected hit ratio. It is presented with its assumptions and never as a measured result.

What we instrument, and with what

Quantity	How we record it	Tools
Per-request latency	Histogram with p50, p90, p99, p99.9 and maximum; never the average alone	Log-linear histograms of the benchmark harness
Throughput	Requests completed per second over the measured period, warm-up excluded	Harness counters; in production, the service’s own metrics (Prometheus or similar)
Work avoided by a cache	Real backend calls versus requests received; hits and misses from the library’s counters	Harness and library counters
CPU and memory	CPU profile and resident memory (RSS) at the end of each run	`/proc`, `perf`, `pprof` in Go; sanitizers (TSAN, ASAN) and `valgrind` during code diagnostics
Errors	Failed responses or saturations during the measured period	Harness counters; in production, the service logs
Variability	At least three repetitions; we publish the median and the min–max range	The harness’s `scripts/summarize.py`

Before and after an intervention

Baseline. Before changing anything, we measure the system as it is, under the same load or the same traffic period, and record date, tool and environment.
One change at a time. We apply one intervention and measure it. Several interventions are measured separately.
Same measurement. We repeat the baseline procedure: same load, same environment, same tools.
Repetitions and range. At least three runs; we report the median and the range. If the baseline range and the result range overlap, we do not claim an improvement.
Declared conditions. Hardware, system, power profile, versions and configuration go in the report. We do not compare figures taken on different hardware.
Delivery. The report follows the template below and includes the raw data.

The first principle is that you must not fool yourself—and you are the easiest person to fool.

Richard P. Feynman, Cargo Cult Science (1974), Caltech commencement address, published in Engineering and Science 37(7). Source.

We test our hypotheses and publish the conditions and limits of every measurement. If the data contradict an idea, we revise it.

Published benchmarks

The protocol, scripts, raw data and environment of each measurement live in the simylresearch/simyl-benchmarks repository. The current results sheet is at Benchmarks. Anyone can repeat the measurement with the README commands and compare against the declared environment.

The same rigor applied to this site

Before every release, this site goes through a lab protocol: a build that fails on any warning; two in-house lints that check links, anchors, metadata, structured data, forms, social images, retired phrases and Spanish–English parity; an automated browser pass that exercises keyboard navigation, light and dark theme, anchors, analytics consent, form validation and behaviour without JavaScript; and an automated check of the published site after each merge and once a month. Every check is recorded with its date.

Templates

Measurement report template (Markdown).
Acceptance criteria template (Markdown).

We use them in every diagnostic and milestone-based project; you are welcome to use them as they are.

Want your system measured this way? Request the discovery call