Request Coalescing: Fixing the Thundering Herd in Analytics

The failure mode

A dashboard query is cached for sixty seconds. Twenty people have it open. At the moment the key expires, all twenty requests miss simultaneously — and all twenty fire the same expensive aggregate.

The cache did not fail. It worked exactly as written. The gap is that a naive cache has no concept of in-flight.

The fix

Track promises, not just values:

async getOrFetch(key, ttl, fetchFn) {
  const cached = this.get(key);
  if (cached !== null) return cached;

  // Someone is already fetching this — join them.
  if (this.inFlight.has(key)) return this.inFlight.get(key);

  const promise = fetchFn()
    .then((data) => { this.set(key, data, ttl); this.inFlight.delete(key); return data; })
    .catch((err) => { this.inFlight.delete(key); throw err; });

  this.inFlight.set(key, promise);
  return promise;
}

The first caller starts the work. Everyone else awaits the same promise. Twenty requests, one query.

The two details that matter

Delete on failure. If the fetch rejects and the promise stays in the map, every subsequent caller gets the same rejection forever. The .catch that removes the key before rethrowing is not optional.

Return the promise, not a copy. Awaiting the same promise is what makes this work. Cloning defeats the purpose.

Where it pays off

Anywhere a cache miss is expensive and concurrent: dashboard aggregates, a report several people open on Monday morning, an API endpoint behind a load balancer.

It costs a Map and about fifteen lines. On a busy analytics instance it is the difference between one query and fifty at every TTL boundary.

Sharing keys across surfaces

A further win: if your REST routes and your AI tool layer use the same cache keys, a question asked in chat that the dashboard already answered is a cache hit rather than a fresh scan. That only works if the key format is shared deliberately rather than by coincidence.