Durable ingestion pipeline

Off by default. The API writes telemetry straight to the store unless you set SENTRINEL_INGEST_PIPELINE=redpanda, in which case ingest appends to a replicated log and a consumer drains that log into ClickHouse.

Why it exists

Without it, ingest is at-most-once with silent loss. The plugin empties its buffers before it posts, and the API writes ClickHouse synchronously before acking — so a storage outage means the batch is refused and already discarded. That is not hypothetical: a full disk took ClickHouse's I/O out from under this service and the telemetry for that window was simply gone.

Two independent fixes ship together, and they cover different failures:

Client retry needs no infrastructure and is enabled for everyone. The pipeline is worth its operational cost once losing a storage window is worse than running one more stateful service.

Turning it on

docker compose --profile queue up -d redpanda

SENTRINEL_INGEST_PIPELINE=redpanda \
SENTRINEL_REDPANDA_BROKERS=localhost:19092 \
  bun run apps/api/src/index.ts

Topics are created on boot: sentrinel.requests and sentrinel.logs, three partitions each, a week of retention, zstd. Errors and traces are not routed through the log — they are written to Postgres by their own routes, and a topic whose consumer cannot write it would fill and never drain.

What it guarantees, and what it does not

At-least-once delivery, deduplicated at the store. Offsets are committed only after the ClickHouse write succeeds, so a crash mid-batch replays rather than loses — and the replay does not duplicate. Every message is written with a deduplication token of topic-partition-offset, which is reproducible: replay the record and it hashes to the same token, and ClickHouse discards the repeat.

Re-reading an entire log wrote 0 duplicate rows.

Two things make that work, and both are easy to undo by accident:

• non_replicated_deduplication_window = 1000 on request_logs and app_logs. Without it the token is accepted and silently ignored.
• One insert per message, not per fetched batch. Fetch boundaries are not reproducible — kafkajs groups by what arrives in a poll — so a token built from them differs on replay. That version wrote 419 duplicates. A message is stable, and is exactly the batch the API accepted, so it is also the right insert size.

The window is a count of recent blocks, so a replay of more than 1000 blocks ago would duplicate again. No crash-and-restart comes near that; a deliberate replay of a week of history would.

A broker outage degrades rather than fails. If the log is unreachable the ingest routes fall through to writing the store directly and say so in the logs. Refusing the batch would lose it for exactly the reason the log exists.

Verified behaviour

With ClickHouse stopped mid-run:

rows before outage: 15327
   ... 382 requests sent while ClickHouse was down ...
   backlog held in Redpanda, TOTAL-LAG 20
rows after drain:   15709      (+382, nothing lost)

Two bugs this found, worth knowing about

JSON has no Date. Rows cross the log as JSON, so every timestamp arrives as a string. The ClickHouse store called .toISOString() on it and threw, which crashed the consumer on its first batch — the pipeline delivered nothing at all. chTs now accepts a string or an epoch.

kafkajs commits on its own. eachBatchAutoResolve and autoCommit both default to true, which advanced the group past batches that were never written: during the outage the consumer retried, crashed, restarted, and came back to a committed offset beyond the backlog. Lag read zero and the rows were gone — precisely the loss this pipeline exists to remove. Both are now off and the offset is committed explicitly, at lastOffset + 1, after the write.

When to bother

Not until one of these is true:

• more than one consumer wants the same stream (alerting, a warehouse export);
• ingest must not block on storage under sustained load;
• replay is a requirement — reprocessing after a fingerprinting change, say;
• roughly >100k events/sec, or a durability SLA you are contractually on the hook for.

Below that, client retry alone closes the loss that actually happens.