Prevent Recurring Prod Incidents

Fix an incident once. Catch it on every change after. Each production incident generates a regression test and a code review rule aimed at its root cause, so the same failure is caught before it ships again.

From incident to guardrail.

Every prod incident automatically generates a regression test and a code review rule targeting its root cause. This prevents recurring prod incidents.

CloudAEyeIncident INC-2041

The incident

Checkout fails for older cartsSEV-1

Checkout returned errors for carts created before multi-currency support. Their currency field is empty.

Root cause

A recent change to checkout/totals.py reads cart.currency without a fallback.

Generated from the root cause

tests/test_totals.pyRegression test
def test_total_for_cart_without_currency():
    cart = make_cart(currency=None)
    assert cart_total(cart).currency == "USD"
Code review ruleTargets the root cause

Code that reads cart.currency must handle carts with no currency. Carts created before multi-currency support have none.

A later pull request that reads the field without a fallback is flagged in review, before it merges.

The same incident
should not happen twice.

A postmortem says what went wrong, but nothing stops the same mistake from shipping again. CloudAEye turns each incident's root cause into a test and a review rule, so the fix stays fixed.

Start from the root cause.

Each incident's root cause is the starting point, so the guardrails target the cause, not the symptom it produced.

  • Starts fromThe root cause
  • TargetsThe cause, not the symptom
  • CoversEvery prod incident
See an example

A regression test for the failure.

A test that reproduces the conditions behind the incident, so CI fails if the bug comes back.

  • TargetsThe root cause
  • RunsWith your other tests
  • FailsIf the bug returns
Explore Unit Tests

A review rule for the pattern.

A plain-language review rule describes the mistake, so every later pull request is checked for the same pattern.

  • Written inPlain language
  • Applies toEvery later review
  • CheckedBefore merge
Explore Custom Rules

Guardrails that add up.

Each incident adds a test and a rule. Over time, reviews and CI cover the failures your team has actually seen.

  • Per incidentA test and a rule
  • Over timeReal failures covered
  • ResultFewer repeat incidents
Explore Learn from PRs

How it works.

It runs automatically for every production incident. There is nothing to trigger.

Read about review rules
  1. A production incident happens.
  2. Its root cause is identified.
  3. A regression test and a code review rule are generated.Both target the root cause.
  4. Later changes are checked.The rule in code review, the test with the rest of your tests.