Alerts & Monitoring
Detect operational failures, policy risks, and cost anomalies before they impact production AI workflows.
Alerts & Monitoring helps teams detect AI failures before they become customer-facing incidents.
Traditional monitoring platforms can detect infrastructure failures.
AITracer monitors operational failures that traditional systems often miss:
- latency degradation
- token spikes
- unusual trace behavior
- policy violations
- workflow failures
- cost anomalies
- model instability
Alert workflow
Alert types
Latency Anomalies
Monitor abnormal response degradation across workflows. Triggers when latency exceeds your configured threshold (default: 5,000ms).
Cost Anomalies
Detect sudden increases in AI spend. Triggers when estimated cost per execution exceeds your configured threshold (default: $1.00).
Token Spikes
Monitor for unusually large token consumption. Triggers when total tokens exceed your configured threshold (default: 50,000 tokens).
Execution Failures
Alerts when a trace status is failed, generating a critical-severity alert with immediate escalation.
Timeout Events
Detects execution timeouts, generating high-severity alerts with the recorded latency.
Governance Violations
Alerts when compliance findings are detected in prompt or response text, including PII, SSN patterns, and medical context.
Verification Failures
Critical alerts when post-persist SHA-256 verification detects a hash mismatch, indicating potential storage corruption or tampering.
Alert severity levels
| Severity | Meaning | Examples |
|---|---|---|
| Critical | Immediate action required | Execution failures, hash mismatches |
| High | Urgent review needed | Timeouts, governance violations, webhook delivery exhausted |
| Medium | Monitor and investigate | Latency anomalies, cost anomalies, token spikes |
| Low | Informational | Below-threshold anomalies |
Alert delivery channels
AITracer can route alerts to your operational teams through multiple channels:
Slack
Configure your Slack webhook URL in the environment variables. Alerts are delivered as formatted messages with severity-color-coded attachments and relevant metadata fields.
PagerDuty
For critical and high-severity alerts, enable PagerDuty integration. Alerts are routed through the PagerDuty Events API with deduplication keys to prevent alert storms.
Operational alerts can be delivered via email for audit trails and team notifications.
In-app
All alerts are always visible in the AITracer dashboard alert feed with filtering by severity, status, and type.
Alert deduplication
To prevent alert fatigue, AITracer deduplicates alerts within a one-hour window. If an alert of the same type already exists as "open" for the same trace, a duplicate is not created.
This ensures your team sees each distinct issue once rather than being overwhelmed by repeated notifications.
Configuring alert thresholds
Alert thresholds are configurable per workspace in Settings → Alert Thresholds:
| Threshold | Default | Range |
|---|---|---|
| Latency (ms) | 5,000 | 100 – 600,000 |
| Cost (USD) | $1.00 | $0.000001 – $1,000,000 |
| Token spike | 50,000 | 1,000 – 100,000,000 |
Health monitoring
The rate limiting system includes self-monitoring with a health endpoint:
GET /api/health/rate-limit
Returns the current state of the rate limiter, including whether it has degraded to in-memory mode (indicating Redis connectivity issues).
Operational benefits
Most AI incidents begin as small anomalies:
- latency slowly increases
- costs quietly spike
- policies begin failing
- workflows degrade over time
Alerts & Monitoring helps teams detect these issues early before they escalate into outages, compliance incidents, or runaway spend.