01 · Premise
Instrumentation is evidence, not an objective
A trace can explain why one request was slow. A metric can show that latency moved across the fleet. A log can preserve the event and trace context. None of them independently says whether users received the service they were promised or whether the team should page someone.
I built the SigNoz SLO Pack to connect those layers. A Node/Express service emits traces, RED metrics, and trace-correlated logs through OpenTelemetry. Three dashboards then move from service health to error-budget state to cost-efficiency signals. Alert rules watch the rate at which the budget is being consumed, and a load generator plus webhook catcher make the response path testable.
The pack is intentionally reusable. Service name, objective, latency threshold, and burn thresholds are knobs rather than assumptions buried in screenshots.
02 · Instrumentation
One request, three complementary views
The demo service measures request count and duration in middleware, emits spans around request and dependency work, and attaches trace identifiers to structured logs. The important part is correlation: an operator should be able to begin with an error-rate panel, find the relevant traces, and arrive at logs carrying the same request context.
The service exposes normal, slow, error, and downstream-failure paths. A continuous mixed load generator keeps the panels populated, but it also gives the verification harness repeatable behaviour to assert against.
Express middleware records count, status class, and duration.
Spans capture request and dependency boundaries.
RED signals aggregate rate, errors, and duration.
Structured events carry service and trace context.
Dashboards and alerts evaluate the same service-level promise.
03 · SLO math
Make the reliability promise computable
For availability, valid requests form the denominator and successful requests form the numerator. A 99% objective allows a 1% error budget over the chosen window. Burn rate normalizes the observed error ratio by that allowed ratio, making different traffic levels and time windows comparable.
A burn rate of 1 means the service is consuming budget exactly as fast as the objective allows. A burn rate of 14.4 means the budget is disappearing 14.4 times too quickly. That is a more actionable signal than a generic CPU or error-count threshold because it is expressed in the currency of the user promise.
04 · Alert design
Fast failures and slow degradation need different windows
A single threshold forces a bad trade-off. Short windows detect incidents quickly but flap on brief noise. Long windows are stable but can wait too long while the budget burns. Multi-window alerting pairs a short confirmation window with a longer window that shows sustained impact.
The pack ships a fast-burn critical rule, a slow-burn warning rule, and a latency-SLO warning. The 14.4 and 6 burn thresholds assume a 30-day objective window and are deliberately exposed for tuning. The point is not that those values are universal; it is that each alert maps to a measurable rate of budget exhaustion.
- Fast burn: page-worthy, large budget loss over a short period
- Slow burn: persistent degradation that deserves intervention before exhaustion
- Latency rule: user-visible response time beyond the chosen p99 boundary
05 · Operator experience
Dashboards should answer progressively better questions
The Service Health dashboard answers what is happening now: request rate, error percentage, p50/p90/p99 latency, status-class throughput, and dependency errors. The SLO dashboard answers whether the service is meeting its promise: availability SLI, latency SLI, remaining budget, and burn rate. The Cost-Efficiency dashboard asks where the service is doing expensive work without producing successful outcomes.
That third view uses a waste index based on latency and error ratio. It is not a financial accounting system; it is a prioritization signal. A route that is both slow and error-prone consumes compute and operator attention without delivering value.
- Service Health → What changed?
- SLO / Error Budget → Does it threaten the promise?
- Cost Efficiency → Where is failed or slow work accumulating?
06 · Verification
If you have never forced the alert, you have not tested the alert
The repository includes a webhook catcher and documented fault knobs. Raising the demo error rate to 90% produces an approximately 18× burn rate, above the 14.4 fast-burn threshold. Raising the maximum slow-path delay beyond six seconds exercises the p99 latency rule. Returning the values to normal verifies recovery as well as firing.
A shell verification harness checks that all three signals arrived, alert rules exist, and the expected repository artefacts are present. This catches a class of demos where the dashboard JSON exists but was never exercised against live telemetry.
# Force fast burn
ERROR_RATE=0.9 docker compose up -d demo-service
# Force the latency objective to fail
SLOW_MAX_MS=10000 docker compose up -d demo-service
# Recover
ERROR_RATE=0.1 SLOW_MAX_MS=3000 docker compose up -d demo-service07 · Reflection
What the setup process revealed
One non-obvious operational dependency is that SigNoz accepts no telemetry until an organization exists. On first run, the administrator must register the organization and restart the collector so it picks up the resulting configuration. A polished dashboard without that step is an empty promise.
The next version should automate dashboard import as fully as alert import, validate query schemas against the installed SigNoz version, and add a longer soak test so burn-rate panels can be compared against independently calculated values. I would also add exemplar links from metrics to traces and record alert delivery latency during drills.
The larger lesson is simple: observability infrastructure is successful when a new operator can reproduce it, inject a known failure, see the expected signal, and understand why the alert fired.