Sentrinel — Production Architecture

Designed for SaaS from day one: multi-tenant, two-store, sampling-aware. Both storage engines are Apache-2.0 licensed — nothing here restricts offering Sentrinel as a paid hosted product.

                     ┌─────────────────────────┐
   customer app      │  @sentrinel/plugin       │  in-process aggregation
   (Elysia.js)  ───▶ │  · exact 1-min counters │  + head sampling of raw logs
                     │  · sampled request logs │  (errors & slow always kept)
                     └───────────┬─────────────┘
                                 │ HTTPS batches (flushInterval)
                                 ▼
                     ┌─────────────────────────┐
                     │  Sentrinel API (Bun)     │  /api/ingest/*
                     │  LogStore abstraction   │  /api/requests|traffic|…
                     └─────┬──────────────┬────┘
              writes/reads │              │ writes/reads
                           ▼              ▼
              ┌────────────────┐   ┌──────────────────────┐
              │ PostgreSQL     │   │ ClickHouse           │
              │ CONTROL PLANE  │   │ TELEMETRY STORE      │
              │ orgs, apps,    │   │ request_logs, spans, │
              │ endpoints,     │   │ app_logs             │
              │ consumers,     │   │ · day partitions     │
              │ alerts, uptime,│   │ · ZSTD codecs        │
              │ 1-min metrics  │   │ · TTL retention      │
              │ (+ optional    │   │ · 1-min MV rollup    │
              │  read replica) │   │   kept 2 years       │
              └────────────────┘   └──────────────────────┘
                           ▲
                           │
                 ┌──────────────────┐
                 │ Next.js dashboard│
                 └──────────────────┘

Why two stores

The measured data shape (see docs/storage sizing): metrics tables grow at ~130 MB/day even with 500 endpoints; request_logs grows at ~13 GB/day at a mere 100 req/s. One table is 99% of the volume and it's append-only, never-updated event data — the exact shape column stores crush (10–30× compression, sub-second scans over billions of rows). Meanwhile apps, alerts and orgs need foreign keys, updates and transactions — the exact shape Postgres is for. SigNoz and Uptrace ship this same split.

The LogStore abstraction (packages/api/src/lib/logstore/)

All firehose reads/writes go through one interface with two implementations:

• TELEMETRY_STORE=postgres (default) — dev and small self-hosted setups. Zero extra infrastructure; retention via batched deletes + BRIN index.
• TELEMETRY_STORE=clickhouse — production. Batched async inserts, TTL retention, materialized per-minute rollups that survive raw-log expiry.

Routes never touch a database directly for logs, so switching stores is an env var, not a migration.

Multi-tenancy

organizations (Postgres) is the tenant root; every app carries org_id, and every ClickHouse row carries org_id as the FIRST column of the sort key (ORDER BY (org_id, app_id, timestamp, …)) so tenant isolation is a sort-key prefix scan, not a filter. This was baked in now because changing a ClickHouse sort key later means rewriting the table. monthlyRequestQuota on organizations is the hook for plan enforcement at the ingest API.

Sampling (the biggest cost lever)

requestLogging.sampleRate in the plugin keeps 1-in-N successful requests; errors and slow requests (≥ slowRequestThresholdMs) are always kept. Counters are aggregated in-process before sampling, so dashboard metrics stay exact at any rate. Every stored row records its effective sample_rate, and the ClickHouse rollup extrapolates (sum(1/sample_rate)), so even log-derived counts stay honest. Verified: a 0.1-rate row aggregates as 10 requests.

Retention

• Raw logs: RETENTION_LOGS_DAYS (default 30). ClickHouse: table TTL with ttl_only_drop_parts → whole-day partition drops, no delete storms. Postgres: 5000-row batched deletes every 6 h.
• Metrics/rollups: RETENTION_METRICS_DAYS (default 400) in Postgres; ClickHouse 1-min rollup kept 730 days.

Read/write separation

Ingest writes always hit the Postgres primary. Dashboard queries use dbRead, which binds to DATABASE_REPLICA_URL when set (managed Postgres providers hand these out; self-hosters can attach a streaming replica) and falls back to the primary otherwise. ClickHouse handles its own read parallelism; add replicas + Distributed tables only at much larger scale.

Scaling ladder

1. Now — single box, TELEMETRY_STORE=postgres. Fine to ~10 req/s.
2. First customers — docker-compose.prod.yml: Postgres + ClickHouse, TELEMETRY_STORE=clickhouse, sampling 0.1–1.0 per plan. Fine to ~1000 req/s on one decent host.
3. Growth — managed Postgres + replica URL; ClickHouse on its own host (or ClickHouse Cloud); API scales horizontally (it's stateless).
4. Big — ClickHouse cluster w/ replication, Kafka/Redpanda in front of ingest for burst absorption, per-org rate limits. The LogStore interface is where a Kafka producer would slot in.

Operations

• ClickHouse schema: bun run --cwd packages/db ch:migrate (idempotent, substitutes retention into TTLs).
• Backups: pg_dump nightly for the control plane; ClickHouse BACKUP TABLE to S3 for telemetry (or accept telemetry as re-ingestable and back up only Postgres — a legitimate early-stage choice).
• Health: GET /health reports the active store and its reachability.