Designing a Real-Time Analytics Pipeline (Without Kafka)

"Real-time" needs a number

Before designing anything, define your freshness budget: how stale can a dashboard number be before it is wrong for its purpose?

• Live visitor counter — a few seconds. This is the only genuinely real-time surface most analytics products have.
• Today's traffic chart — 30–60 seconds is imperceptible.
• Weekly trends — hourly is fine.

Almost every over-engineered analytics pipeline comes from applying the first budget to all three. A 60-second budget is dramatically cheaper than a 1-second budget, and for most panels nobody can tell the difference.

The single-node pipeline

You do not need Kafka, Flink, and a streaming warehouse to serve real-time analytics for a normal website. The shape that works:

Tracking script
   │ POST /api/track  (sendBeacon)
   ▼
Ingest endpoint ──▶ in-memory queue ──▶ batch INSERT ──▶ PostgreSQL
                                                            │
                                              keyset sync (every ~5s)
                                                            ▼
Dashboard ◀── SSE / poll ◀── analytics API ◀────────────  DuckDB

Every arrow here is a place where you choose between latency and cost. Being explicit about that is the whole design.

Ingestion: make it cheap and non-blocking

The tracking endpoint should do as little as possible synchronously — validate, enqueue, return.

app.post('/api/track', (req, res) => {
    const event = validate(req.body);          // cheap, synchronous
    if (!event) return res.status(204).end();  // never leak validation detail
    queue.push(event);                          // in-memory
    res.status(204).end();                      // respond immediately
});

Return 204 immediately. The browser does not need a result, and holding the connection open while you write to a database makes your ingest latency a function of your database's worst moment.

On the client, use navigator.sendBeacon so events survive page unload:

navigator.sendBeacon('/api/track', JSON.stringify(payload));

sendBeacon is queued by the browser and delivered even as the page is being torn down — which is exactly when your most interesting events (exit page, time on page, final scroll depth) fire.

Batching: the single biggest lever

Writing one row per request is what turns a modest traffic spike into an outage. Accumulate and flush on whichever comes first — size or time:

const BATCH_SIZE = 500;
const FLUSH_MS = 1000;

setInterval(async () => {
    if (!queue.length) return;
    const batch = queue.splice(0, BATCH_SIZE);
    try {
        await bulkInsert(batch);
    } catch (err) {
        if (queue.length < MAX_QUEUE) queue.unshift(...batch);  // bounded retry
        logger.error({ err, size: batch.length }, 'flush failed');
    }
}, FLUSH_MS);

A dual trigger matters: size alone means a low-traffic site never flushes; time alone means a spike builds an unbounded batch.

Backpressure, honestly handled

An in-memory queue must be bounded. When the database is slow and events keep arriving, you have exactly three options, and you must pick one deliberately:

1. Drop new events when the queue is full. Preserves history, loses the present.

2. Drop oldest events. Preserves the present, loses history.

3. Spill to disk and drain later. Preserves everything, adds a failure mode.

For analytics, dropping is usually acceptable — losing 0.5% of pageviews during an incident is not a correctness problem. What is not acceptable is an unbounded queue, which converts a slow database into an OOM kill and loses everything.

Whichever you choose, count what you dropped and expose it as a metric. Silent data loss is how dashboards start lying without anyone noticing.

Sync to the read layer

Writes land in Postgres; reads come from DuckDB. The sync interval is your freshness budget for everything except the live counter — a 5-second keyset-cursor sync means dashboard numbers are at most 5 seconds stale, which is well inside any reasonable budget.

Use a keyset cursor on the event id and checkpoint after every batch, so a restart resumes exactly where it stopped without duplicating or skipping rows.

The live counter is a special case

"Visitors in the last 5 minutes" does not need to go through the whole pipeline. Keep it in memory or in Redis as a sliding window, updated at ingest:

liveWindow.add(event.visitorId, Date.now());
// count = distinct visitor ids seen in the last 300s

This is the one surface where sub-second freshness is genuinely visible to the user, and it is cheap precisely because it is separated from the durable path.

Delivering updates to the browser

Server-Sent Events are the right default. One-directional, plain HTTP, automatic reconnection built into the browser, and no protocol upgrade:

app.get('/api/realtime/stream', (req, res) => {
    res.writeHead(200, {
        'Content-Type': 'text/event-stream',
        'Cache-Control': 'no-cache',
        Connection: 'keep-alive',
    });
    const timer = setInterval(async () => {
        const snapshot = await getLiveSnapshot(req.siteId);
        res.write(`data: ${JSON.stringify(snapshot)}\n\n`);
    }, 5000);
    req.on('close', () => clearInterval(timer));
});

The req.on('close') cleanup is mandatory. Without it every disconnected client leaves a timer running forever, and the leak only becomes visible under real traffic.

WebSockets are the wrong tool here — you gain bidirectionality you do not need and take on connection management you would rather avoid. Plain polling every 10 seconds is also perfectly respectable and simpler still.

Frequently asked questions

What counts as real-time?

Whatever number you commit to. "Real-time" without a freshness budget is unfalsifiable; pick a target such as five seconds end to end and design against it, because that number decides every other choice.

Do I need Kafka for a real-time pipeline?

Almost certainly not at one-node volumes. An in-process queue with batched writes handles thousands of events per second on modest hardware. Kafka earns its operational cost with multiple consumers and cross-machine durability.

What is the single biggest performance lever?

Batching writes. Amortizing per-statement overhead across many rows typically improves ingestion throughput by an order of magnitude and costs only the batch interval in added latency.

How should backpressure be handled?

Explicitly, with a bounded buffer and a stated drop policy. An unbounded queue does not remove backpressure, it converts it into memory growth and eventually an out-of-memory crash.

Why is the live visitor counter handled separately?

Because it needs freshness, not history. It reads recent state directly rather than waiting for the sync into the analytics store, which is why it updates in seconds while historical charts lag a cycle.

When you do need the heavy machinery

Reach for Kafka and a streaming engine when you actually have:

• Multiple independent consumers of the same event stream.
• Ingest beyond what one node can absorb — sustained tens of thousands of events per second.
• Replay requirements — reprocessing history after a logic change.
• Cross-datacenter durability guarantees.

Below that, a bounded in-memory queue, batched inserts, a keyset sync, and SSE will serve a real-time dashboard on hardware that costs less per month than the streaming platform's documentation is long. InsightsTrack runs this exact pipeline — see the architecture.