Sentrinel — what each page does

A tour of every screen: what it answers, where its data comes from, and what you need to have configured for it to show anything. Written against the shipped build — everything here works today.

Two data paths feed the whole product:

• Aggregated metrics — counters the plugin computes in-process and flushes every 30s, stored in Postgres. Exact regardless of sampling, cheap to query over long windows. These drive Traffic, Performance, Consumers, Errors.
• Raw events — individual request logs, application logs and spans, stored in ClickHouse in production. Detailed but sampled. These drive Request logs, the Trace waterfall, and Apdex.

When a page looks empty, it is almost always because one of those two paths isn't enabled — the tables below say which.

Apps

Answers: which services report to me, and is anything on fire right now?

Cards for each registered app with requests, error rate, p95, and consumer count for the selected window, plus a sparkline. This is the app switcher: the app selected here scopes every other page.

Apps are created two ways — explicitly (Create app) or automatically the first time a plugin authenticates with a key for a name it hasn't seen.

Traffic

Answers: how much am I serving, and which endpoints carry it?

A time series of requests/errors/success plus a per-endpoint table sorted by volume, with error rate, average latency and bytes transferred. Drag on any chart to zoom into a window; every page respects that selection.

Click an endpoint for its own view: requests and latency over time, a response time histogram, status-code breakdown, and — if request logging is on — the most recent requests, slowest requests, and errors for that endpoint specifically.

Those three lists come from raw logs. With only metrics ingest enabled the charts fill in and the lists stay empty; that is expected, not a bug.

Source: aggregated metrics; endpoint lists from raw logs.

Errors

Answers: what is failing, on which endpoints, for whom?

Client (4xx) vs server (5xx) errors over time, grouped by status code and endpoint, with occurrence counts and affected-consumer counts. Good for "we're returning a lot of 422s" — for "why", go to Issues.

Source: error ingest.

Issues

Answers: what are the distinct bugs, and are they getting better?

Every error occurrence is fingerprinted from its type and normalized stack, so a crash firing 50,000 times is one row, not 50,000. Each issue carries a title, culprit (the top frame in your code), occurrence count, first/last seen, and affected consumers.

Issues have a lifecycle: resolve one, and if it recurs it comes back automatically marked as a regression rather than quietly reopening. Bulk resolve/ignore from the list.

Each issue can be assigned to a team member, and the list filters by assignee — including Unassigned, which is usually the one you are hunting. A list of bugs becomes a list of work once each has a name on it.

Open an issue for its occurrence timeline, affected endpoints and consumers, and recent raw occurrences with full stack traces. Every occurrence carries Open the request → and its trace id, so a bug reaches the exact request that produced it rather than dead-ending at a stack trace.

Source: error ingest with stack traces. Thrown exceptions are captured with their real stack — the plugin hooks the error path before the framework turns them into a response, which is what makes grouping work.

Performance

Answers: is it fast, and where is the time going?

p50/p95/p99 over time, a latency histogram, slowest endpoints, and Apdex — a single 0–1 satisfaction score. Apdex counts each request as satisfied (≤ your threshold), tolerating (≤ 4×), or frustrated, and weights them.

Set the threshold per app, or override it per endpoint — a report generator and a login endpoint should not be judged the same way.

Source: metrics for the percentiles; raw logs for Apdex, which needs per-request timings. No request logging means no Apdex.

Consumers

Answers: who is calling me, how much, and who is having a bad time?

Per-client request volume, error rate, and average latency. A consumer is whatever consumerIdentifier returns — an API key name, tenant id, or app name.

Open one for the whole trail, in one place:

This is the "customer says the API is broken" screen.

How the trail is assembled

Every signal stores its own consumerIdentifier rather than being joined back through the request that produced it. That is deliberate: raw logs, replays and requests expire on separate retention clocks, so a trail built by join would quietly shorten as the requests behind it aged out — and a partial answer to "everything this user did" is worse than no answer.

Two identities feed it, and both now land in the same place:

• Backend — whatever consumerIdentifier returns in the plugin.
• Web and mobile — identify() in the browser SDK (userId), or distinctId from the Flutter/native SDKs. Both are accepted.

Set the same value on both sides and one person's backend calls, browser sessions and recordings line up under a single record.

Source: consumer metrics; per-consumer lists, sessions, replays and logs read by identifier.

Request logs

Answers: show me this exact request.

A searchable, filterable feed of individual requests: method, path, status, latency, consumer, timestamp, and where the call came from — each row carries the caller's country as a flag. Filter by status class, method, path, log level, and now country or host:

/api/requests?country=TZ
/api/requests?host=api.example.com

Go live streams new requests as they arrive (SSE), carrying the same fields as the list.

Open one for the full picture:

Related is the one people underuse: it reconstructs what a client was doing around a failure, which is usually where the cause is.

Requires requestLogging.enabled. Bodies and headers additionally require logRequestBody / logResponseBody / logRequestHeaders. Client IP and country need a proxy in front of your app that forwards them — see PLUGIN.md for which headers are read.

Logs vs Request logs

Two different things with confusingly similar names, so plainly:

They are two views of the same moment. A request lists exactly what it logged (its Logs tab); a log record opens the request it was written during. Neither is a subset of the other — a request that logged nothing still has a row, and a background job's logs belong to no request at all.

Logs

Answers: what did the code actually say, and about whom?

A log explorer built around fields rather than text. The message is the stable, groupable part ("Checkout rejected"); the attributes are what you filter on (reason=insufficient_funds, tier=enterprise, consumer=mobile_app).

The level tiles double as filters — click WARN for warnings and above. The category dropdown filters a whole subsystem (payment selects payment.checkout and payment.fraud). Clicking any attribute value in a row adds it as a filter, which is how you get from "something is wrong" to "it is wrong for this one customer on this one endpoint" without writing a query. Go live tails matching records as they arrive.

Fields opens the vocabulary of your own telemetry: every attribute key in play with its commonest values, each clickable as a filter. You should not have to already know that checkout attaches tier and fraud attaches riskBand.

Click a record to open it in full — three tabs:

• Record — the message, every attribute, and whether this message is a one-off or a pattern
• Same request — everything else that request logged, in order, with this record highlighted
• Connections — the request being handled (with the business context the handler attached), the trace and span, and the issue if that request failed

From there, "every log in this trace" pulls the records from every service that took part, and "open the waterfall" jumps to the spans.

The reverse direction works too — every request has a Logs tab listing exactly what it logged.

Requires logging: { } in the plugin (or logCapture for plain console output). See PLUGIN.md.

Traces

Answers: where did the time actually go inside this request?

A list of traces, and a proper waterfall for each: spans nested by parent, all on one timeline, coloured by kind (SERVER / CLIENT / DB / INTERNAL).

Click any span for its duration, self time, status, and full attributes. Self time is the number to look at — a 271 ms span that spent 245 ms waiting on a child isn't the problem, its child is. The panel calls out the slowest span by self time so you can start there.

Root spans link through to the request log entry that produced them, and traceparent propagation means a call into another instrumented service continues the same trace rather than starting a new one.

Requires traceSpan()/traced() in your code, or an OTLP exporter. Without them you get the root HTTP span only.

Resources

Answers: is the host healthy?

CPU and memory (RSS, heap total, heap used) over time, reported by the plugin on every metrics flush. Useful for separating "the code got slower" from "the box ran out of memory".

Each chart draws the average as a line and the min–max range as a shaded band. The band is the point: several replicas averaging 40% CPU read as healthy whether they are all at 40% or one is pinned at 100%, and only the spread tells them apart.

Overlay traffic plots request volume on a second axis, so you can see whether a CPU spike followed load or caused it.

Source: automatic — no configuration.

Databases

Answers: which query is costing me, and what is everything waiting on?

Postgres monitoring, fed by a collector you run next to the database. Sentrinel never connects to it — the collector holds the credentials and stays on your side of the network, and query literals are stripped before anything is sent.

Queries ranks by share of total database time, not by mean duration. That ordering is the whole point: a 4ms query called a million times costs more than a 900ms one called twice, and sorting by "slowest" hides it. The collector's own polling queries appear here too, deliberately — you should be able to see what the observer costs.

Activity samples pg_stat_activity once a second, which is what makes wait events and blocking usable: the lock you care about is held for 200ms and is invisible at a ten-second cadence. A row reading Lock / transactionid with a blocking PID is one transaction stuck behind another.

Metrics covers connections against max_connections, cache hit ratio, rollback rate and deadlocks.

Tables lists sizes and dead-tuple ratios, and turns them into advisories — bloat, missing indexes, and indexes larger than the table they serve. Small tables are exempt from the missing-index rule, because a sequential scan of 200 rows is the right plan.

Click a table for its columns, its write mix, and every index with its definition, its size and its scan count side by side — which Postgres itself will not show you, and which is the whole judgement: size is a permanent tax on every write to the table, scans are what it earns back. From that pair come the findings worth acting on: an index nobody has ever scanned, one that went quiet weeks ago, one whose columns another index already leads with, and one left invalid by a failed CREATE INDEX CONCURRENTLY. Primary keys and unique indexes are never called unused — they enforce a constraint, which is worth their size whether or not anything reads them.

Health covers the four failures that stop a Postgres database without warning anyone first, all of which are knowable days ahead and none of which show up on a dashboard of averages: transaction ID wraparound, sequence exhaustion, an abandoned replication slot filling the disk, and checkpoints firing on write volume rather than the clock. It also shows vacuums in flight with their progress, without which a long vacuum and a stuck one look identical.

Errors is what actually went wrong, read from the server log and grouped by SQLSTATE. The counters tell you a deadlock happened; only this says which transactions, on what table, running which statement. Query and error text is shown in full by default, values included, because that is what makes a failure reproducible. Where it should not be — regulated data, or a dashboard audience wider than the database's — SENTRINEL_MASK_QUERIES=true strips literals on your database host before anything is sent. Grouping is unaffected either way.

Disk and backups cover the two failures Postgres cannot report on itself. Free space is measured from the filesystem holding the data directory — and reported as unavailable, rather than guessed, when the collector is not on the database host. Alongside it is the growth rate, because "82% full" is a number people ignore and "82% full, nine days left" is a date. WAL archiving is checked for whether it works rather than whether it is switched on: a failing archive_command leaves a database serving traffic normally with a backup chain that has been broken for days.

Alerts come with the database. Four rules are created the moment it is added — wraparound past half the limit, any inactive replication slot, connections past 85%, a sequence past 70%, disk under 15% free, WAL archiving failing — because a rule you have to know to configure prevents very few outages. They carry no notification channel until you attach one. A collector that stops reporting puts its rules into "not reporting" rather than resolving them to green: zero on a > rule looks identical to healthy, and it is not.

Config & Security audits settings, roles, grants and extensions, and says what each finding costs you and how to fix it. Two rules keep it honest: it never reports a check it could not run as passing, and it distinguishes TLS being enabled from TLS being used — a server that offers encryption while every client connects in plaintext looks compliant on a settings page.

Product answers who uses the thing. People, active now, where they are on a live map, what pages and features they touch, funnels with their drop-off, retention cohorts, and web vitals by country, device and release.

The number that matters most is on Issues, though: sort by users affected rather than occurrences. A retry loop firing ten thousand times is loud and hurts one person; a checkout crash firing twenty-five times hurts twenty-five people. Occurrence count measures volume, not harm.

Product alerts cover the failures that are not errors — active users falling off a cliff, which is often the first sign of an outage, since a request that never arrives cannot fail. See PRODUCT.md.

Without pg_stat_statements you still get everything except the Queries tab. The collector says so at startup rather than refusing to run. The same applies to the two optional grants: sequence checks and log-based errors each report which grant is missing rather than showing an empty page.

Source: @sentrinel/pg-collector — a read-only role holding pg_monitor, no superuser.

Uptime

Answers: is it up from the outside?

HTTP checks on an interval, with uptime percentage, response time, and an incident timeline. Complements the plugin: the plugin can only report while your process is alive, so a check from outside is what catches a hard down.

Alerts

Answers: tell me before a customer does.

Threshold rules on error rate, p95 latency, or request count over a window, with Slack / Discord / generic webhook delivery. Alerts resolve themselves and send a recovery notice.

Each alert keeps its firing history, so you can see whether it is a real signal or noise that needs its threshold moved.

Set up channels first; an alert with no channel only changes colour in the UI.

SLOs

Answers: how much error budget is left this month?

Availability and latency objectives over a rolling window (default 30 days), with attainment, remaining error budget, and burn-down.

The difference from an alert: an alert fires on a moment, an SLO tracks a promise. "99.9% availability" means about 43 minutes of downtime a month — the budget tells you how much you have spent.

Cron monitors

Answers: did my nightly job actually run?

The thing uptime checks cannot tell you. Each monitor gets a check-in URL; the job pings it on completion. If a ping doesn't arrive within its interval plus grace, the monitor is marked missed and notifies your channels.

Long jobs can report ?state=start, a ?duration=, or an explicit ?state=fail, giving you run duration history and distinguishing "failed" from "never ran".

Metrics

Answers: what did this cost us?

The numbers only your application knows — tokens, spend, revenue, queue depth, jobs retried — pushed with count(), gauge() and histogram() from the plugin. Pick a series, choose how to aggregate it, and split it by any label you recorded.

Everything the page offers comes from the data itself: the metric list, the available aggregates and the label splits are all discovered, so there is nothing to configure before it is useful.

The aggregate follows the kind, because picking the wrong one silently answers a different question — summing a gauge, or reading the "last" of a counter, is a number that looks fine and means nothing.

This is the half tracing cannot do. A span says a model call took 900 ms and failed twice; it cannot say you spent $41 on that model today, because nothing in a span is a number you asked to be summed.

Requires TELEMETRY_STORE=clickhouse. See PLUGIN.md for the API and the cardinality rules.

LLM cost

Answers: what is the AI spend, and who is spending it?

Cost, call count, token usage and latency broken down by model, by client, and by endpoint, priced from a built-in table.

Populated from GenAI spans — gen_ai.request.model plus gen_ai.usage.*_tokens — so any OTel-instrumented LLM call is priced automatically.

Release health

Answers: is this build worse than the one it replaced?

Crash-free sessions and crash-free users, per release, over a chosen window.

Error counts cannot answer that question. A release that doubles usage doubles its errors while being no less stable, and one that crashes on launch reports almost nothing because nobody gets far enough to hit anything else. The denominator is what makes it a measurement: one session per app launch.

Two rates, because they answer different things:

• Crash-free sessions — how often the app falls over.
• Crash-free users — how many people lived through it.

One user stuck in a crash loop tanks the first and barely moves the second, and which you are looking at changes what you do about it.

Abnormal is counted separately from crashed. A force-quit, an OOM kill and a flat battery all end a session without a crash report; counting those against the release would make the metric worthless. The SDK marks its session the moment a fatal error is written, so the next launch knows which it was.

Fed by the Flutter SDK, which reports sessions automatically once you pass a release to Sentrinel.init(). Without a release every build is "unknown" and a regression is invisible.

Session replay

Answers: what was the user actually doing when it broke?

A DOM recording of the seconds leading up to an error — the click, the form state, the screen they were looking at. Recorded in the browser by the SDK, buffered in memory, and uploaded only when an error fires, so a session that never fails costs nothing.

Open one and it plays back with a scrubber. The list shows the page, what triggered it, duration, event count and size.

Everything is masked by default. Text renders as blocks and input values are never recorded; you opt individual elements in with data-sentrinel-unmask. Nobody turns replay on expecting to ship customer details to a vendor, so that switch points the safe way.

Requires TELEMETRY_STORE=clickhouse — recordings are too large for the Postgres store, and ingest refuses them rather than dropping them silently. Retention is its own knob (RETENTION_REPLAY_DAYS, default 14). See BROWSER.md.

Dashboards

Answers: the handful of numbers I check first thing.

Saved arrangements of views the product already computes — requests, error rate, latency percentiles, top endpoints and clients, CPU, memory, open issues, LLM spend. Compose one in Settings → Custom dashboards, then open it here.

A board belongs to the app that was selected when it was saved, and shows under that app only — like everything else an app has. Only the widgets a board actually uses are fetched, so a two-chart board makes two calls.

Status pages

Answers: telling customers before they ask.

A public page at /public/status/<slug> showing component health from the selected app's uptime checks, plus incidents you post. Each page belongs to one app and publishes that app's uptime only. No authentication — deliberately: a status page behind a login is useless during the outage it exists for.

SQL console

Answers: the question no dashboard anticipated.

Read-only SQL over the telemetry tables, with guardrails: a single SELECT or WITH, allowlisted tables only, no system catalogs, a hard row cap, and no access to users or sessions — so it can never read a password hash or steal a session token.

A query reads the selected app's data and nothing else, whatever it says: every table it can name is narrowed to that app before the query runs, so there is no app_id to remember and none to fake. Schema-qualified names (public.x, system.x) and functions that run SQL text or reach outside the database are refused.

Settings

Eight sections, split between what belongs to you and what belongs to the organization.

Copying an API key

Keys used to be visible only in the seconds after you created one, so losing one meant minting a replacement and redeploying whatever used it. The keys table now has a Copy key button per row.

Worth being clear about the trade-off: keys are stored in plaintext, so this widens who can read them and when rather than changing how they are kept. Any signed-in member of your org can copy any of your org's keys at any time. If that is not what you want, revoke and rotate rather than sharing dashboard access.

Which key to issue

Every key is bound to one integration, chosen when it is generated (What is this key for?), and refused on every other surface:

The kind and the key's last-used time are on the keys table, so "is the collector on the new replica sending?" is a glance rather than a query. Keys issued before kinds existed are server keys and keep working unchanged.

Renaming a key, or changing what it is for

Edit on a key row does both, and they are different operations underneath.

A name is only a label, so it changes in place and the secret is untouched — whatever is deployed keeps working.

Changing what the key is for issues a new secret and revokes the old one. That is not a limitation to work around: a key carries its kind in its prefix (snt_mobile_…, snt_mcp_…), and authentication requires the prefix and the stored kind to agree, so a key that changed kind cannot keep the same string. The dialog says so before you confirm. The old key is kept as a revoked record rather than deleted, so what it did stays in the audit trail.

Trials and billing

New organizations start on a 14-day trial (SENTRINEL_TRIAL_DAYS; set it to 0 on a self-hosted deployment, where there is nobody to bill). Organizations that existed before trials shipped are never put on a clock — they stay active.

A countdown sits at the foot of the sidebar once a trial is running, and turns red if the org is blocked.

When a trial lapses, billing goes past due, or the org burns its monthly quota, the dashboard content is blurred behind a card explaining which of the three it is. The sidebar stays clickable — someone who has just been blocked needs Settings, and trapping them on one screen turns a payment prompt into a support ticket. The card's Choose plan button goes straight to Settings → Billing.

The org owner is exempt from the blur: the one person who can settle billing or raise a quota is never locked out of their own Settings. Other roles still see the gate, so a blocked org stays visibly blocked without paralysing its own admin.

Telemetry keeps arriving for SENTRINEL_INGEST_GRACE_DAYS (default 7) after that block. Cutting data the instant a card fails would mean that by the time someone pays, the window they needed to look at is gone. After the grace period ingest answers 402 — see TROUBLESHOOTING.md for what that body says.

Nothing already collected is ever deleted for non-payment.

Platform admin

Only visible to the product owner and any operator listed in SENTRINEL_ADMIN_EMAILS, and only useful if you run Sentrinel rather than merely use it.

One table of every organization on the instance: plan, billing status and why it is blocked, trial end, usage against quota, apps, members. Manage opens per-org controls with three one-click actions — Require payment now, Extend trial 14 days, Mark as paid — plus the underlying fields.

Changes reach the customer's dashboard on their next load, and ingest within about 15 seconds.

To everyone else these routes answer 404 rather than 403: a 403 would confirm the surface exists and that they merely lack the role.

What is not here

Being straight about the edges:

• SMTP. Email alerts work through Resend or any HTTP mail provider, but speaking SMTP directly does not. Put a relay in front of your mail server and point the generic provider at it.
• RUM, product analytics, feature flags. Out of scope. Session replay is supported; product analytics is a different product.
• On-call scheduling. Point the webhook channel at PagerDuty or Better Stack rather than rebuilding it.
• The SQL console routes per table. Telemetry tables are queried in ClickHouse when that is the configured store and in Postgres otherwise, so a raw-log query answers from raw logs rather than silently from the rollups. What it will not do is join across the two stores in one statement.