Operations runbooks

The owner/admin Operations page evaluates tenant-scoped signals every time it loads and refreshes every 30 seconds. Start with the alert's trace and correlation IDs. Never paste credentials, raw callback bodies, customer content, email addresses, or phone numbers into an incident record.

API errors

1. Confirm whether failures share one operation and deployment version.
2. Follow the trace into the application audit using the correlation ID.
3. Separate 4xx caller errors from 5xx application/dependency errors.
4. If a release caused the regression, stop rollout and use the documented change-control rollback. Do not mutate production data to make a probe pass.
5. Resolve only after a complete healthy evaluation window; a quiet window is insufficient evidence.

API latency

1. Compare p95 by operation and identify whether provider or database spans dominate.
2. Check database health and current job backlog before increasing limits.
3. Confirm the issue with persisted production-shaped inputs; fixture-only performance is not evidence of recovery.
4. Roll back the offending release or isolate the dependency, then verify a healthy 15-minute window.

Job failures

1. Inspect the job name, attempt, backend, and correlation trail.
2. Load the referenced persisted resource; never trust a workspace ID supplied by an external payload.
3. Classify the error as retryable, terminal, permission, or invalid content.
4. Repair the cause before replay. Keep the original idempotency key and do not publish or trigger a paid provider action without explicit approval.
5. Confirm the resource and application audit reached the intended tenant.

Provider errors

1. Group by provider, status, and operation; do not inspect or log tokens.
2. For 401/403, revalidate current account ownership, scopes, and token state.
3. For 429, respect the provider's retry window instead of bypassing limits.
4. For 5xx or network failures, preserve idempotency and ambiguity recovery before retrying a write.
5. Confirm recovery with a non-destructive provider probe or an explicitly authorized sandbox action.

Webhook failures

1. Confirm the exact public callback URL and raw-body signature algorithm.
2. Treat payload tenant hints as untrusted; route only through signed state or a globally unique provider identity.
3. Check replay identity and monotonic state handling before reprocessing.
4. Never disable signature validation to clear the alert.
5. Verify a signed callback reaches one tenant and records an application audit before resolving.

Escalation record

Record the alert rule, first/last trigger, deployment version, trace and correlation IDs, impact, owner, mitigation, verification, and follow-up. Use sanitized references only. Security or privacy incidents follow the incident response and data-breach decision process; provider outages follow the relevant provider status and support channel.