Alerts and notifications

Watch a metric every minute and hear about it in Slack, email, a signed webhook or PagerDuty.

On this page

Tracing tells you what happened when you go looking. An alert is what goes looking for you: a condition Tracehatch measures every minute, and the channels it tells when the answer crosses a line.

#What a rule is

A rule is one measurement, one comparison and one window, narrowed to the slice of your project you care about.

Evaluated: Once a minute, per rule
Window: 5 minutes to 24 hours
Rules: 50 per project
Severity: Info, warning or critical

The metrics are failure rate, error count, median and p95 duration, cost per run, spend, token volume, and "no data" for a project that has gone quiet. Each compares above or below a threshold, or change against the window before it — "p95 is more than 50% worse than the previous half hour" rather than any particular number of milliseconds.

The scope is what turns a project-wide number into a useful one: an environment, an agent, a model, a provider, a tool, a tag, or an error type. Not every pairing is measurable, and the editor says so rather than saving a rule that quietly watches nothing — a duration scoped to a tag, for instance, has no timings behind it.

#Creating one

Open Alerts → Rules

Three templates start you off: failure rate above 10% over 15 minutes, a p95 latency spike of 50% over half an hour, and daily spend up 50% against the day before.

Pick what it measures

The metric, the comparison and the window. The editor charts the last 24 hours with your threshold drawn across it, so you can see whether the rule would have been firing all week before you save it.

Narrow it

An environment — production, by default — and optionally one agent, model, provider, tool or tag.

Choose channels

Any of the workspace's notification channels. A rule with no channel still raises alerts, and they still reach the in-app inbox; a channel is only how an alert leaves Tracehatch.

#What happens when it fires

The condition going true opens an alert, announces it to every channel on the rule, and shows it at the top of the dashboard. The condition staying false for one clear window resolves it on its own, and the channels hear that too.

Cooldown — 15 minutes by default. Inside it a fresh breach is recorded on the rule but raises nothing, so a flapping condition does not become a flood.
Acknowledging says a person is on it. It stops nothing, because nothing further is being sent; it is there so the next person can see it is handled. Any member can acknowledge or resolve, while managing rules stays with owners and admins.
Resolving by hand closes the alert and starts the cooldown.
Editing what a rule measures closes its open alert, as a manual resolution, and restarts the cooldown. The evaluator only auto-resolves an alert the current condition disagrees with, so an alert raised at a threshold of 10 would otherwise stay open for ever once you moved the threshold to 25. Renaming a rule, or changing its severity or channels, leaves the alert alone.

Each alert keeps a snapshot of the metric around the moment it fired, so the chart on the alert page still draws long after the minute-level aggregates behind it have been pruned.

#Channels

Channels belong to the workspace, at Settings → Integrations, and any rule in any project can use them.

Type	What it needs
Email	Up to 20 recipient addresses
Slack	An incoming webhook URL on `hooks.slack.com`
Webhook	An HTTPS URL; we generate the signing secret
PagerDuty	An Events API v2 routing key

Every message carries the same sentence — "Failure rate is 12.4% (threshold 10%) over 15 min" — a link to the alert, and a link to the traces from exactly the window that raised it. PagerDuty triggers and resolves against the alert's own id, so an incident closes itself when the condition clears.

A webhook channel is signed. Each request carries Tracehatch-Signature: t=<unix seconds>,v1=<hex>, where the hex is HMAC-SHA256 over <t>.<raw body> using the secret shown once when you create the channel. Verify it in constant time and reject a timestamp more than five minutes old.

Credentials are encrypted at rest, never returned by the API, and never written to a log. A webhook channel's signing secret is displayed exactly once; a channel has no rotation, so if you lose it, create a new channel.

That signature is the same contract an outbound webhook endpoint uses, so one verifier handles both. The difference is what they are for: a channel is one alert rule's destination, while an endpoint subscribes to event types across the whole workspace, keeps a 30-day delivery log, retries for about a day and can rotate its secret with a 24-hour overlap.

#The inbox

An alert also reaches the bell in the header, for every member of the workspace, whether or not the rule has a channel — a channel is how an alert leaves Tracehatch, and the inbox is where it stays.

Plans: Every plan
Kept: 90 days
Badge: Unread only
Full page: Account → Notifications

Four things arrive there today: an alert firing, an alert resolving, a teammate accepting an invitation, and a workspace passing 80% or 100% of its monthly trace allowance. The two alert notifications reach every role, viewers included, because anyone who can read the alerts page wants to know one fired. The other two reach owners and admins only, since membership and spend are theirs to act on — and the person who has just joined is left out of the notice that they did.

The bell, alert rules and notification channels are included in Free preview. Free workspaces can receive alerts, accepted-invitation notifications and usage notices, with the same per-type notification preferences as other plans.

The badge counts unread notifications and nothing else: onboarding progress keeps its place as a pinned row inside the popover rather than joining the count, so one number never stands for two things, and a workspace with steps left but nothing unread shows a plain marker instead. The popover filters All and Unread; the full page at Account → Notifications adds date headings and its own paging. Marking read is one notification or all of them, and turning a type off stops new notifications rather than hiding the ones already there.

Reading the inbox over HTTP needs a signed-in session rather than a token — see Your inbox.

#Through the API

Rules and alerts are project-scoped; channels are workspace-scoped. Reading needs alerts:read, everything else alerts:write.

Rules live at https://api.tracehatch.com/api/v1/projects/{projectId}/alert-rules.

GET and POST there for rules, GET /projects/{projectId}/alerts for what has fired, and POST /projects/{projectId}/alerts/{alertId}/acknowledge or /resolve to act on one. Channels live at /workspaces/{workspaceId}/notification-channels. Full shapes are in the HTTP API reference.

Production health, releases and historical trends shipped alongside alerting and have their own page: Health, releases and trends.

#Not built yet

Anomaly detection without thresholds, muting and snoozing, on-call routing, and incident grouping are all later work. The inbox has no push, desktop or digest delivery, and no dismiss or undo beyond marking a notification read.