← BACK TO DISPATCH
Dispatch ・ ESSAY ・ ・ 2 MIN READ

Test the outage where everything fails at once

I tested each component failing on its own. The registry down, the scorer down, the loop detector down. All green. Then I pictured a database outage taking down two at once, and realized I had never tested the failure that actually happens.

Test the outage where everything fails at once

I was proud of my fallback logic, and that pride is what nearly shipped the bug.

A production routing agent leans on at least three things that can fail independently, the capability registry, the scoring model, and the loop detector. A naive build lets any one of them crash the whole routing call. So I built a degradation hierarchy instead, registry dies, fall back to a hardcoded default set, scorer dies, use uniform weights, loop detector dies, keep routing without loop protection and log a warning. Then I tested each one. Registry down, green. Scorer down, green. Detector down, green. A partial registry failure using whatever capabilities survived, green. A clean row of single-failure tests, every fallback firing exactly as designed. Done, I thought.

It was not done, because real outages do not politely fail one component at a time. A database outage takes down the registry and the scorer in the same instant, because they sit behind the same connection. Failures correlate. And a system that handles each failure alone but has never been asked to handle two at once will crash the day two arrive together, which is precisely the day you most need it standing.

So I wrote the test I had been avoiding without realizing it. Patch all three to raise the moment they are touched, total simultaneous failure, then route a task and assert two things, the call does not propagate an exception, and the result comes from the documented fallback set. If the router raises instead of degrading down to its floor, that test fails loudly, on my machine, instead of quietly at 3am on a pager.

That total-failure test is the one I would keep if I could keep only one. The single-component tests prove each fallback exists. Only the all-at-once test proves they compose, that the floor actually holds when the whole floor below it gives way together.

Graceful degradation you only tested one failure at a time is not graceful degradation. It is a guess about a correlated outage you never ran.

Part of a series. Start here: A green test suite proves less than you think