Benchmarks

New measurements, taken on 18 October 2026 on the machine described below, of two in-process cache libraries with a single-flight protocol: gateway_cache (Go) and cpp-cache (C++20). They are not the 2023 tests of the full architecture described in 100K requests per second, and they are not production traffic: each library is measured alone, under synthetic load. The general procedure is in How we measure; the protocol, scripts and raw data are in simylresearch/simyl-benchmarks.

In three conclusions

1. Single-flight works in both libraries. On a cold start, 64 workers requesting 100 uncomputed keys produced exactly 100 backend calls in both: nobody recomputes what someone else is already computing.
2. On the hit path, cpp-cache delivers more and uses less memory. With one worker it serves about 3.7 million requests per second against 1.8 million for gateway_cache; from 4 to 16 workers both settle around 1.6–2.1 million, with a lower p99 and between a third and a half of the memory in cpp-cache.
3. When the backend dominates, the library stops mattering. With frequent misses and a 1 or 2 ms backend, both land between roughly 19,000 and 24,000 requests per second: the backend sets the throughput, and an in-process cache does not change that.

What these figures do not show is in “What they show, and what they do not”, below: no network, no serialization, no other hardware.

What was measured

• A trace of 4,000,000 requests per scenario, generated from a fixed seed with a Zipf distribution, replayed identically by both libraries.
• W workers walk the trace from different offsets; every request calls the “get or compute” operation with the key as a string.
• The simulated backend counts its calls and sleeps for the scenario’s latency; the time it really slept is published (“real backend”).
• Per-request latency goes into a log-linear histogram (≈6 % resolution); throughput is requests completed per second across all workers.
• Unmeasured warm-up, 10 s measured (3 s for the cold start), 3 repetitions alternating libraries. Each cell is the median; the min–max range of throughput shows the variability. The p99.9 percentiles and the maximum of each run are in the repository’s RESULTS.md.

Where and when

• Machine: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics (8 cores, 16 threads, L3 16 MiB); 58.5 GiB of memory; Ubuntu 24.04.5 LTS, kernel 6.8.0-146-generic. A laptop with the “balanced” power profile and the powersave governor (amd-pstate-epp), on mains power.
• Tools: Go 1.24.4; GCC 13.2.0; CMake 3.28.3. Harness at commit 2614b33.
• Date: 18 October 2026. Total wall time of the protocol: 501 s; 42 runs (14 configurations × 3 repetitions), with 0 errors in the raw data.
• It is a laptop in normal use, not an isolated server: the variability across repetitions is published as a range. The figures are not extrapolated to other hardware or to real traffic.

Results

Hits: throughput by worker count

100,000 keys that fit in the cache, zero-cost backend, 60 s TTL and a 4 s warm-up to cover the whole trace: the hit path and lock contention from 1 to 16 workers.

Misses with LRU eviction

1,000,000 keys, Zipf 0.8, capacity 10,000, 1 ms backend, 16 workers: miss-heavy traffic with constant LRU eviction. Backend calls versus misses show how many concurrent misses on the same key were coalesced.

Cold start with 64 workers

100 keys, uniform distribution, 64 workers, 50 ms backend, no warm-up: every worker asks for keys nobody has computed yet. With single-flight the backend is called exactly 100 times; without it, up to 64 times per key.

Hot and cold keys with a 1 s TTL

200,000 keys, Zipf 1.0, 2 ms backend, 1 s TTL, 16 workers: both libraries renew the TTL on every hit, so popular keys never expire while keys requested less than once per second are recomputed. A mix of microsecond hits and 2 ms recomputations.

What they show, and what they do not

• Single-flight works in both libraries. In the cold start, 64 workers asking for 100 uncomputed keys produced exactly 100 backend calls in each; the maximum latency (≈ 51 ms) is the wait for a single 50 ms call.
• Hit path. With one worker, cpp-cache serves about 3.7 million requests per second and gateway_cache about 1.8 million. From 4 to 16 workers both level off (≈ 1.9–2.1 million and ≈ 1.6–1.7 million): each library’s global lock bounds the scaling. Under contention the median of cpp-cache rises to 3.6 µs while that of gateway_cache stays at 0.4 µs, but the p99 of cpp-cache with 16 workers is 53 µs against 279 µs. Resident memory with 100,000 entries: ≈ 52 MB against 122–154 MB.
• When the backend dominates. With a million keys, capacity 10,000 and a 1 ms backend, both deliver ≈ 19,000 requests/s with 16 workers: throughput is set by workers ÷ (miss fraction × backend latency), not by the cache. Backend calls equal misses in both: with that many distinct keys, two workers rarely miss on the same key at the same time, so single-flight barely comes into play.
• Hot and cold keys. With a 1 s TTL and a 2 ms backend, both land at ≈ 22,000–24,000 requests/s: recomputations of cold keys dominate and the p99 of both is the backend itself (≈ 2 ms).
• Variability. In the hit scenarios the range across repetitions reaches 10–25 % depending on the case: this is a laptop in use. In the backend-dominated scenarios the range is below 1 %.
• What they do not show. No HTTP layer, serialization, network or real traffic; nothing about other hardware. Simulated latencies are 1 ms or more because Go’s time.Sleep rounds shorter values up to ≈ 1 ms. The 2023 figures (100K requests/s of the full architecture) are a different measurement and are not compared with these.

Reproduce it

git clone https://github.com/simylresearch/simyl-benchmarks
cd simyl-benchmarks
scripts/run.sh

Requirements are in the repository README. Raw data of this measurement: results/2026-10-18-ryzen7-pro-7840u/raw/results.jsonl; full environment: results/2026-10-18-ryzen7-pro-7840u/environment.json.

Want a measurement like this of your system, before and after a change? Request the discovery call