I spent the next six hours explaining why the thing that fixed itself had broken everything else.
That was the night I stopped trusting my own loop.
We had spent months getting to that architecture. Three rewrites. One production incident that does not come up in polite conversation. The core idea was clean: watch the metrics, detect drift, fire a remediation, verify recovery, close the loop. A heartbeat that healed itself.
When it worked, I would sit there and watch dashboards in a kind of quiet disbelief. A spike would appear and the system would catch it. Pull it back. In seconds. No human, no ticket, no 2 AM scramble. There is a particular dopamine that comes from watching a machine do exactly what you built it to do, silently, without you having to lift a finger. I got addicted to that feeling.
That is where the danger lives.
The night everything broke, the loop did exactly what it was designed to do. A service drifted. The loop detected it. The loop identified the fix. The loop applied it.
To all sixteen instances. Simultaneously.
The fix was wrong. Not catastrophically, not obviously. Wrong the way subtle is worse than obvious, correcting one metric while quietly degrading two others. The kind of wrong that needs scale to reveal itself.
Sixteen instances, all applying the same mistake, all at the same time. All reporting healthy.
By the time I was back at my laptop, the original drift was gone. In its place was something much harder to trace. A consistent wrongness, distributed evenly, applied with the full efficiency of a machine that never second-guesses itself.
Later, staring at the timeline, I found the thing I had missed: the loop was treating the remediation procedure as already proven. It had been proven in testing. It had been proven in past incidents. But that is not the same as proof it would behave correctly, at scale, right now.
A canary gate makes you prove it again. Every time.
Apply the fix to a small slice first. One instance, maybe two. Give it real time, not theoretical time. Watch not just the metric you were chasing but the adjacent ones. The downstream ones. The ones that do not show up in the obvious dashboard. When the canary comes back clean, let the loop go wide.
It adds seconds. At most minutes. It does not add more time than a human incident response. What it removes is the ability to be fast in the wrong direction.
The redesign took two days. Not because canary gating is hard, but because you have to think carefully about what "watching" actually means for your system. Which signals matter for this specific remediation? How long does the canary have to hold before you trust it? How small does the slice need to be to contain the blast if the fix turns out to be wrong again?
Those are not universal answers. They are specific to your failure modes, your topology, your acceptable risk. We got some of them wrong the first time around. That is fine. Getting the canary gate tuned incorrectly costs you a delayed fix or a false alarm. Getting the loop wrong without a canary gate costs you a six-hour incident and a postmortem you will write at sunrise.
One of those failures fits in a sprint. The other one stays with you.
Speed and confidence are not the same thing.
A self-heal loop is fast. That is the point. But fast without observation is not confidence, it is just velocity. There is a real difference between a system that heals quickly because it verified the fix worked on a small slice first, and a system that heals quickly because it never stopped to check.
Build the loop. Automate the healing. Let the machine take the 1:47 AM alert. But give it a gate. Make it prove the fix before it goes wide. Make it watch a small piece of the world first and tell you the world is still okay. Only then let it run.
The machine should be faster than a human. It does not need to be less careful than one.