Outbound webhooksPro

An alert channel tells a person. A webhook tells a program: Tracehatch posts a signed JSON envelope to a URL you own whenever something you subscribed to happens, and keeps a log of every attempt.

#What you get Pro

Endpoints

25 per workspace

Retries

10 attempts over ~22.7 h

Timeout

10 seconds per attempt

Delivery log

30 days

An endpoint belongs to the workspace, not to a project, so one integration serves every project you have. You can narrow it to a single project, and — inside that project — to a single environment, which is the usual way to keep staging traffic out of a production incident tool.

#Adding an endpoint

#Events

Nine types can be subscribed to. Each arrives as its own envelope; there is no batching and no wildcard, so an event type added to Tracehatch later will not start arriving at an endpoint you did not update.

#The envelope

Every delivery is POST with Content-Type: application/json and the same top-level shape, whatever the type. The three tenancy fields are hoisted out of the body so you can read workspace in the same place every time; project and environment are null for a workspace-level event such as member.joined.

{
  "id": "evt_01j9xk3m4np6q7r8s9t0v1w2x3",
  "type": "trace.failed",
  "created_at": "2026-09-20T12:00:03.120Z",
  "workspace": { "id": "ws_…", "slug": "acme" },
  "project": { "id": "proj_…", "slug": "support-agent" },
  "environment": { "id": "env_…", "slug": "production" },
  "data": {
    "trace": {
      "id": "run_…",
      "name": "answer-question",
      "status": "error",
      "started_at": "2026-09-20T12:00:00.000Z",
      "ended_at": "2026-09-20T12:00:02.900Z",
      "duration_ms": 2900,
      "span_count": 7,
      "failure_count": 1,
      "cost_usd": 0.0142,
      "url": "https://tracehatch.com/acme/support-agent/traces/run_…"
    },
    "error": {
      "type": "TimeoutError",
      "message": "upstream did not respond",
      "span_id": "span_…"
    }
  }
}

id is the event's own id and is stable across every retry and every redelivery of that event, which makes it the right key to deduplicate on. data carries the type's own fields and nothing else.

#Headers and the signature

POST <your url>
Content-Type: application/json
User-Agent: Tracehatch-Webhooks/1
Tracehatch-Event-Id: evt_…
Tracehatch-Event-Type: trace.failed
Tracehatch-Signature: t=1758369603,v1=5f1c…

Tracehatch-Signature is a comma-separated list. t is the Unix timestamp in seconds at which the request was signed, and each v1 is

hex( hmac_sha256( secret, `${t}.${raw_request_body}` ) )

over the exact bytes of the body — sign the string you received, never a re-serialised parse of it.

To verify: reject the request if t is more than 300 seconds from your own clock, then accept it if any v1 matches your computed digest, compared in constant time.

import { createHmac, timingSafeEqual } from "node:crypto"

const REPLAY_WINDOW_SECONDS = 300

export function verifyTracehatch(rawBody, header, secret) {
  const parts = header.split(",").map((part) => part.trim())
  const timestamp = Number(
    parts.find((part) => part.startsWith("t="))?.slice(2)
  )
  if (!Number.isFinite(timestamp)) return false
  if (Math.abs(Date.now() / 1000 - timestamp) > REPLAY_WINDOW_SECONDS) {
    return false
  }

  const expected = Buffer.from(
    createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex")
  )

  return parts
    .filter((part) => part.startsWith("v1="))
    .map((part) => Buffer.from(part.slice(3)))
    .some(
      (given) =>
        given.length === expected.length && timingSafeEqual(given, expected)
    )
}

#Rotating the secret

Rotate secret issues a new one and shows it once. The previous secret keeps signing for 24 hours, and deliveries in that window carry two v1= values — one for each secret. That is why the check above accepts any match: you can deploy the new secret whenever it suits you inside that day without dropping a delivery, and a receiver that has not cut over yet keeps working until it does.

After 24 hours the old secret stops signing. There is no way to read either secret back, so if you lose the new one before deploying it, rotate again.

#Retries and what "failed" means

Answer with any 2xx as soon as you have stored the event; do the work afterwards. A response is expected within 10 seconds.

Ten attempts span roughly 22.7 hours. Timeouts, 408, 429 and any 5xx are retried. Everything else stops immediately, because a later attempt cannot change the answer:

• Any other 4xx marks the delivery failed and does not retry.
• A 3xx is a failure. Redirects are not followed — point the endpoint at the final URL.
• 410 Gone marks the delivery failed and switches the endpoint off, which is the polite way to tell us to stop.

A delivery that uses all ten attempts is exhausted. Nothing picks it up again on its own; it waits in the log for you to redeliver it.

An endpoint that fails for 3 days with no success in between is disabled automatically, and the workspace's owners and admins get a notice in their inbox. Re-enabling it clears the failure record, so the three days start again from then.

#The delivery log

Each endpoint keeps 30 days of deliveries: the exact envelope that was sent, the attempt count, your status code, the first 1 KB of your response body and the latency.

Redeliver sends a stored envelope again. It creates a new entry rather than restarting the old one, so the original attempt stays readable, and it is signed afresh with a current timestamp so it passes your replay-window check. The event id is unchanged, which is what lets your deduplication treat it as the same event.

#Limits worth knowing

• Deduplicate on Tracehatch-Event-Id. Delivery is at-least-once. A response we never see — a timeout on your side after you committed, a connection dropped mid-answer — is retried.
• Order is not guaranteed. Retries interleave with new events. Use created_at and the resource's own fields rather than arrival order.
• There is no per-endpoint rate limit. If you subscribe a busy project to trace.*, you will receive one delivery per run.
• There is no dead-letter feed. exhausted deliveries sit in the log until somebody redelivers them.
• Deleting or disabling an endpoint is always available, on every plan, even after a downgrade that removes the Pro feature — so a workspace can always stop what it started.
• Webhook delivery stops on Free. Endpoint settings stay saved. Queued deliveries are marked failed without sending; requests already in flight may still arrive. Upgrading restores new events for enabled endpoints (trace events within about a minute), but does not automatically replay stopped deliveries. Creating, editing, rotating, testing and redelivering need Pro.

#Through the API

Everything above is available on the HTTP API under /workspaces/{id}/webhooks, with the token scope webhooks:write for the writes and projects:read for the list and the delivery log.