SLOs that change decisions, not dashboards.

Many teams have a reliability dashboard. Fewer have a reliability target that changes what they do on an ordinary Tuesday. The difference is whether the number is connected to decisions: when to page someone, when to slow down feature work, whether a risky deploy can go out this afternoon.

A service level objective is that connection. It states how reliable a user-facing behavior needs to be over a window of time, and so it implies an error budget: the amount of unreliability the business has agreed to accept. Spending the budget is fine; that is what it is for. Running out of it is the signal to change priorities.

Start with what users do, not with what your infrastructure happens to expose. CPU usage, pod restarts, and queue depth are useful for diagnosis, but no user has ever experienced CPU. Pick the few journeys that matter most, such as signing in, searching, checking out, or receiving a webhook, and define for each an indicator that is a ratio of good events to valid events.

Measure as close to the user as practical. A service that reports 100% success while the load balancer in front of it times out is measuring the wrong thing. And be explicit about what counts as valid: requests rejected because the client sent bad input are usually excluded, so that one misbehaving client cannot spend everyone's budget.

The target should be lower than instinct suggests. A hundred percent is the wrong target for almost everything: users cannot tell 99.99% from 100% through their own networks and devices, and chasing it freezes change. Base the first target on what the service has actually delivered in recent weeks and on what users have complained about, then adjust it.

The arithmetic is what makes the target useful. A 99.9% objective over 30 days allows 0.1% of valid events to fail, which for a time-based view is about 43 minutes of complete unavailability in the window. Framed as a budget, the question changes from "are we up?" to "how fast are we spending what we agreed we could lose?"

Threshold alerts on an error rate are either too sensitive, paging on a two-minute blip that cost a sliver of the budget, or too slow, missing a steady leak that drains it over days. Burn rate fixes both. It is how fast the budget is being spent relative to the pace that would use it up exactly at the end of the window. A burn rate of 1 spends the whole budget in 30 days; a burn rate of 14.4 spends 2% of it in an hour.

Google's SRE workbook recommends combining several burn rates, each checked over a long window and a short one. The long window ensures enough budget has been spent to matter. The short window, about a twelfth of the long one, makes the alert stop soon after the problem does.

In Prometheus, record the error ratio for each window once, then write the alerts against the recorded series:

groups:
  - name: checkout-slo
    rules:
      # Failed checkout requests ÷ all checkout requests, per window.
      - record: checkout:error_ratio:rate5m
        expr: |
          sum(rate(http_requests_total{route="/checkout", code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{route="/checkout"}[5m]))
      # ...the same rule for the 30m, 1h, 6h, and 3d windows.

      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          checkout:error_ratio:rate1h > (14.4 * 0.001)
          and
          checkout:error_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: Checkout is spending its 30-day error budget 14x too fast.
          runbook: https://runbooks.example.com/checkout-slo

      # The 6h / 30m page at a burn rate of 6 follows the same shape.

      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          checkout:error_ratio:rate3d > 0.001
          and
          checkout:error_ratio:rate6h > 0.001
        labels:
          severity: ticket

These rules count only server errors. To bring the latency threshold into the same indicator, count requests slower than 1.5 seconds as failures too, using the histogram bucket at that boundary. Two practical notes. Low-traffic services make ratios jumpy (one failure in twenty requests is a 5% error rate), so add synthetic traffic, lengthen the windows, or alert on absolute counts. And every page should link to a runbook whose first question is "what changed?"

An SLO only changes decisions if the consequences were agreed before they were needed. Write an error budget policy with the product owner, not just the engineers, and sign it off while nothing is on fire. Google publishes an example policy that makes a good starting point.

## Checkout error budget policy

Objective: 99.9% of valid checkout requests succeed within 1.5 s,
over a rolling 30-day window.

While budget remains
- Releases follow the normal process.

When a single incident consumes more than 20% of the budget
- A blameless postmortem is written within five working days,
  with at least one priority action to prevent recurrence.

When the budget is exhausted
- Only reliability and security fixes are released until the
  rolling window is back within the objective.
- The next planning cycle includes the top postmortem actions.

Exceptions need sign-off from the product owner and the engineering lead.

This is where the SLO earns its keep. Feature work and reliability work stop competing on opinion: the budget decides, and the decision was made in advance by the people accountable for both.

1. Pick the single journey that would hurt most if it failed.
2. Define its indicator as good events over valid events, measured near the user.
3. Set the target from recent history, not aspiration.
4. Add the fast and slow burn-rate alerts, each linked to a runbook.
5. Agree the error budget policy with the product owner.
6. After a month, check that the alerts fired when users were hurt, and only then.

Once one SLO is changing decisions, adding the next is routine. Ten objectives nobody acts on are worth less than one that people do. Setting up this loop is part of how we approach cloud and DevOps engagements.