100K Requests per Second: Single-Flight Caching, Hash Routing, and a Custom Scaler

TL;DR — An in-process cache with a single-flight protocol, combined with hash-based routing, a shared second cache tier, and horizontal scaling, was validated in 2023 up to 100,000 requests per second in Kubernetes, in controlled environments, with p99 latency under 500 ms. This post presents the architecture, the concurrency model, and the operational lessons learned.

What was measured, and where. The load figures (up to 100K RPS, p99 < 500 ms) come from tests run in 2023 in controlled environments using the Go implementation (gateway_cache), not from real production traffic. That implementation has, however, been running in production for more than five years on the sports data platform of Genius Sports. cpp-cache is the C++20 implementation of the same protocol; its 2026 measurements, taken separately in a declared environment, are in Benchmarks; we do not attribute the 2023 figures to it. The architecture, what was measured and who did it are summarized in the experience entry Cache and architecture for a sports betting API.

The Core Problem: Thundering Herd in Caches

Consider any expensive operation — a database query, an external API call, a complex computation. Without coordination, multiple threads requesting the same key simultaneously will each trigger the expensive operation independently.

This is the problem that cpp-cache solves by implementing a Thundering Herd Shield. To understand why this mechanism is critical for system stability beyond simple data storage, we recommend reading our previous post: When a Cache Does More Than Just Cache.

System Architecture

The overall system has three layers working in concert. The diagram shows the architecture the figures were obtained with, where each worker’s L1 cache is gateway_cache (Go); cpp-cache implements the same design in C++20 and fits in the same place in the architecture.

flowchart TD
    Client["🌐 Clients\n(browsers, mobile, services)"]
    LB["⚖️ Load Balancer / API Gateway\n(AWS ALB · NGINX · Envoy)"]
    Router["🔀 Hash Router\n(consistent hashing ring)"]

    subgraph GW ["Gateway Layer — N worker processes / pods"]
        GW1["Worker 1\nL1 cache · shard A"]
        GW2["Worker 2\nL1 cache · shard B"]
        GWN["Worker N\nL1 cache · shard Z"]
    end

    L2["🧱 Shared L2 cache\n(ElastiCache Redis)"]
    Backend["🗄️ Origin Backend\n(database · microservice · external API)"]

    Client -->|HTTP/gRPC| LB
    LB -->|any routing| Router
    Router -->|key hash → shard| GW1
    Router -->|key hash → shard| GW2
    Router -->|key hash → shard| GWN
    GW1 -->|L1 miss| L2
    GW2 -->|L1 miss| L2
    GWN -->|L1 miss| L2
    L2 -->|L2 miss| Backend

The load balancer distributes connections; as long as both ring membership and the worker process are stable, the hash router ensures a given key always lands on the same worker process and therefore the same L1 cache instance. When workers are added or removed, consistent hashing remaps only a fraction of the keys (on the order of 1/N), not all of them; those keys start cold on their new worker. And because the L1 cache lives in process memory, a worker that restarts keeps its keys but comes back with an empty cache. This routing affinity converts random cache misses into a near-deterministic hit rate, bounded by each instance’s capacity (LRU eviction) and by entry expiration (TTL).

Layer 1 — cpp-cache: Architecture and Concurrency Model

cpp-cache is a header-only C++20 in-process cache built on Aleph-w data structures. It implements the same concurrency model designed in C++ in 1998, which has been running in production for more than five years in its Go version, gateway_cache.

Internal Components

Three key implementation choices drive performance:

• OLhashTable (open-addressing with linear probing) gives cache-friendly lookups with no pointer chasing.
• Short global mutex + per-entry mutex with condition variable — the global mutex protects only the table and the LRU list during microsecond-level structural operations (lookup, insert, reorder), hits included; it is never held while a value is being computed. Waiting for an in-flight computation happens on the entry’s own mutex, so only threads asking for the same key wait on each other.
• Lock-free statistics via std::atomic<size_t> counters — recording metrics adds no contention on any mutex.

Entry State Machine

Each cache entry transitions through a well-defined state machine that is the heart of thundering herd prevention. The COMPUTING state is the shield: while a solver runs, every other thread requesting the same key finds COMPUTING and waits on the per-entry condition variable rather than launching a duplicate computation. When the solver finishes, all waiters are notified atomically and share the result.

The Go Production Implementation: gateway_cache and gw_cache

The concurrency protocol described above predates both implementations: it was designed in C++ in 1998. In 2020, its Go version, gateway_cache, was published, which has been running in production on the sports data platform of Genius Sports for more than five years; in 2023 its capacity was validated up to 100K RPS in controlled environments. Two Go repositories carry this lineage:

• gateway_cache — the Go library published in 2020 by Genius Sports and used in production; the author’s copy, with later development, is at lrleon/gateway_cache.
• gw_cache — a standalone variant with a cleaner package boundary.

Choosing Between Go and C++

Both implementations follow the same concurrency protocol, whose Promela/SPIN formal model is in the repository (see the previous post and the formal verification with SPIN). In practice: start with Go if your service is already Go-based — the integration is trivial and the production track record is long. Consider C++ if you are embedding the cache in a native binary or if GC pauses are unacceptable for your latency targets. Avoiding GC helps reach sub-millisecond p99, but does not guarantee it on its own: load, the critical path, memory, synchronization, and infrastructure all matter.

Layer 2 — Hash Routing

Routing by key hash transforms a fleet of independent caches into a logically partitioned, collectively large cache. Without it, each worker caches the full key space independently — memory is wasted and backend load multiplies with fleet size.

With 150 virtual nodes per physical node, the key distribution across nodes deviates less than 5% from the mean for typical key distributions. Note that a balanced distribution of keys does not guarantee a balanced distribution of work: a very hot key sends all of its traffic to a single worker, and that case needs to be monitored separately.

Effective Capacity Multiplication

Without routing, N pods with capacity C provide effective cache capacity of roughly C (each pod caches the same hot keys). With hash routing, capacity approaches N × C.

Layer 3 — The Predictive Scaler

The most common Kubernetes Horizontal Pod Autoscaler (HPA) setup scales on CPU or memory, and in our case it reacted too late to fast traffic spikes. HPA also supports custom and external metrics, such as RPS; the comparison in this post is against HPA configured on CPU/memory, not against every possible configuration. We use a custom scaler.

How it works: PID-like feedback + trend-based anticipation

The scaler combines two distinct mechanisms. A closed-loop (PID-like) controller adjusts the replica count based on the gap between observed requests per second (RPS) and available capacity; on its own it is reactive and does not predict demand. The anticipatory part is a separate component that extrapolates the trend. Its effectiveness rests on two factors:

1. Deterministic Capacity Knowledge: We have empirically measured exactly how many RPS a single pod can sustain without degrading latency.
2. Anticipatory Logic: By analyzing historical trends and the current velocity of traffic growth, the scaler “forecasts” the required number of replicas and provisions them before the traffic hits the peak.

This reduces the “cold-start” latency spikes typical of purely reactive scaling, and the likelihood of the fleet being undersized when a spike arrives.

Lessons from Production

1. Hash routing is the force multiplier — key affinity is more impactful than any cache sizing increase.
2. Negative TTL + circuit breaker — essential pair; one prevents missing-key floods, the other bounds backend degradation.
3. Scale on the right signal — in our case, scaling on RPS and trend worked clearly better than scaling on CPU. With HPA, that means configuring custom metrics instead of relying on CPU/memory.
4. Failover is the weakest link — Kubernetes reschedules pods without considering cache state; the cold-cache spike is the highest operational risk.
5. Warmup period matters — ramp traffic linearly to new pods over 2–5 minutes.

Summary

In the 2023 controlled-environment tests, the combination reached 100K RPS with p99 under 500 ms and over 95% L1 hit rate. The origin receives far less than the L1 miss rate because most of those misses are served by L2 (Redis): about 0.25% of total traffic reaches the origin.

At SIMYL Research we design and build systems like this, from high-performance architecture to Kubernetes operations: high-performance backend and API development and technical consulting on performance and architecture. If your backend is struggling under load, the performance and architecture diagnostic is a first step with a fixed scope and a reference price.

Tags: C++20 caching high-performance distributed-systems hash-routing kubernetes aws backend-engineering