Durable ingestion pipeline

On whenever a broker answers. Ingest appends to a replicated log and a consumer drains that log into ClickHouse; with no broker reachable it writes the store directly instead. Nothing has to be configured for either — running a broker is what turns the log on.

Both halves can fail, and the API fails over between them in both directions:

Why it exists

Without it, ingest is at-most-once with silent loss. The plugin empties its buffers before it posts, and the API writes ClickHouse synchronously before acking — so a storage outage means the batch is refused and already discarded. That is not hypothetical: a full disk took ClickHouse's I/O out from under this service and the telemetry for that window was simply gone.

Two independent fixes ship together, and they cover different failures:

Client retry needs no infrastructure and is enabled for everyone. The pipeline is worth its operational cost once losing a storage window is worse than running one more stateful service.

Turning it on

Run a broker. That is the whole step:

docker compose --profile queue up -d redpanda

SENTRINEL_REDPANDA_BROKERS=localhost:19092 \
  bun run apps/api/src/index.ts

The API probes the broker on boot and prefers the log the moment one answers — including a broker that appears after boot, which is when the consumer is started. SENTRINEL_INGEST_PIPELINE only pins the preference order; it does not turn failover off, because a pinned preference that refused the surviving path would be dropping telemetry to honour a setting.

Which half is carrying traffic is reported on /health under ingest, so a pipeline running on one leg is visible rather than inferred:

{ "ingest": { "policy": "auto", "using": "redpanda", "redpanda": "up", "direct": "up" } }

Topics are created on boot: sentrinel.requests and sentrinel.logs, three partitions each, a week of retention, zstd. Errors and traces are not routed through the log — they are written to Postgres by their own routes, and a topic whose consumer cannot write it would fill and never drain.

What it guarantees, and what it does not

At-least-once delivery, deduplicated at the store. Offsets are committed only after the ClickHouse write succeeds, so a crash mid-batch replays rather than loses — and the replay does not duplicate. Every message is written with a deduplication token of topic-partition-offset, which is reproducible: replay the record and it hashes to the same token, and ClickHouse discards the repeat.

Re-reading an entire log wrote 0 duplicate rows.

Two things make that work, and both are easy to undo by accident:

• non_replicated_deduplication_window = 1000 on request_logs and app_logs. Without it the token is accepted and silently ignored.
• One insert per message, not per fetched batch. Fetch boundaries are not reproducible — kafkajs groups by what arrives in a poll — so a token built from them differs on replay. That version wrote 419 duplicates. A message is stable, and is exactly the batch the API accepted, so it is also the right insert size.

The window is a count of recent blocks, so a replay of more than 1000 blocks ago would duplicate again. No crash-and-restart comes near that; a deliberate replay of a week of history would.

A broker outage degrades rather than fails. If the log is unreachable the ingest routes fall through to writing the store directly and say so in the logs. Refusing the batch would lose it for exactly the reason the log exists.

Failover, and why it is a breaker

A failed path is marked down and skipped for a cooldown rather than retried on the next batch. That is not an optimisation detail — it is the difference between a degraded path and an outage.

Production ran for five days with its broker container deleted. Every single ingest call walked the producer's retry ladder before falling through to the store it could have used immediately, so a component that was merely absent became a latency problem for the component that was fine.

So: the first batch to hit a dead path pays for the discovery, and nothing after it does. What notices the recovery is a background probe, never a customer's request. Two consequences worth knowing:

• The log is probed before it is published to. Discovering an absent broker by publishing costs seconds; a probe with no retries answers immediately.
• When both paths are inside their cooldown the batch is refused straight away. A fast 5xx and a slow one have the same outcome — the plugin still holds the batch — and the fast one costs nobody anything.

Verified behaviour

With ClickHouse stopped mid-run:

rows before outage: 15327
   ... 382 requests sent while ClickHouse was down ...
   backlog held in Redpanda, TOTAL-LAG 20
rows after drain:   15709      (+382, nothing lost)

Two bugs this found, worth knowing about

JSON has no Date. Rows cross the log as JSON, so every timestamp arrives as a string. The ClickHouse store called .toISOString() on it and threw, which crashed the consumer on its first batch — the pipeline delivered nothing at all. chTs now accepts a string or an epoch.

kafkajs commits on its own. eachBatchAutoResolve and autoCommit both default to true, which advanced the group past batches that were never written: during the outage the consumer retried, crashed, restarted, and came back to a committed offset beyond the backlog. Lag read zero and the rows were gone — precisely the loss this pipeline exists to remove. Both are now off and the offset is committed explicitly, at lastOffset + 1, after the write.

When to bother

Not until one of these is true:

• more than one consumer wants the same stream (alerting, a warehouse export);
• ingest must not block on storage under sustained load;
• replay is a requirement — reprocessing after a fingerprinting change, say;
• roughly >100k events/sec, or a durability SLA you are contractually on the hook for.

Below that, client retry alone closes the loss that actually happens.