Set cost alerts that fire on real anomalies, not noise
A flat “alert if today costs 50% more than average” either spams you on a naturally spiky workload or stays silent through a real runaway. The right threshold depends on how much your daily LLM cost already varies. This tool turns your average and standard deviation into sigma-based thresholds with a known false-positive rate.
How it works
Daily cost is treated as roughly normally distributed around your average. A threshold set k standard deviations above the mean fires only when cost exceeds:
threshold = average + k × standard_deviationThe further out you set k, the rarer false alarms become. Under a normal distribution the one-sided exceedance probabilities are about 16% at 1σ, 2.3% at 2σ, and 0.13% at 3σ — which the tool converts into expected false positives per month from your noise tolerance.
Why not just use a flat percentage?
A flat “alert if cost is 50% above average” feels intuitive but creates two failure modes. For a workload with high natural variance — say, a service where Monday processing is 3× quieter than Friday — the 50% threshold fires every week and trains the team to ignore it. For a workload that is extremely steady, the same threshold might not fire even during a serious runaway, because the problem manifests as a 20% daily creep over two weeks rather than a single large spike.
Sigma-based thresholds sidestep both problems by calibrating to the workload’s actual variability. The threshold is set relative to how much the cost naturally bounces, not an arbitrary percentage of the mean.
Worked example
Suppose your service averages $12.00/day in LLM costs with a standard deviation of $2.50 (based on the last 30 days):
- 1σ threshold: $12.00 + 1 × $2.50 = $14.50 — fires about 16% of days (~5 per month). Too noisy for alerting.
- 2σ threshold: $12.00 + 2 × $2.50 = $17.00 — fires about 2.3% of days (~0.7 per month). Good for a warning.
- 3σ threshold: $12.00 + 3 × $2.50 = $19.50 — fires about 0.13% of days (~0.04 per month, roughly once every two years under normal conditions). Good for an escalation page.
If a runaway prompt loop drove costs to $35 on a given day, both the 2σ and 3σ thresholds would fire — you’d get the warning and the page without having been flooded with false alarms all month.
Tiered alert structure
A two-tier setup combines early warning with actionable escalation:
| Level | Threshold | Expected fires | Action |
|---|---|---|---|
| Warning | 2σ above mean | ~0.7/month | Investigate during business hours |
| Critical | 3σ above mean | ~0.04/month | Page on-call, block new traffic if needed |
Wire the warning to a Slack notification; wire the critical to PagerDuty or equivalent. This keeps on-call burden minimal while still catching genuine runaways before they compound.
Getting the standard deviation
Export 14–30 days of daily LLM cost from your provider’s usage dashboard (OpenAI, Anthropic, and Google all provide this). Paste the daily figures into a spreadsheet and use STDEV() (Excel/Sheets) or compute it in Python with statistics.stdev(). Update the measurement monthly as your traffic patterns evolve.
Tips for effective alerting
- Two tiers beat one. Use a 2σ warning to investigate and a 3σ page for emergencies, so on-call only wakes for genuine runaways.
- Re-measure variance periodically. As traffic grows your average and standard deviation both shift — recompute monthly so thresholds stay tuned.
- Pair with a hard cap. Sigma thresholds catch relative spikes; an absolute daily ceiling also protects you from a slow, steady climb that never trips a variance-based alert.
- Watch weekday vs. weekend patterns separately. If your workload is much quieter on weekends, a blended 30-day average will produce a threshold that is too high on weekdays (misses problems) and too low on weekends (generates noise).