Stop writing examples, start stating invariants

Every example test I write is drawn from my own imagination, which is the one place my blind spots are guaranteed not to be.

That is why first implementations fail on edge cases their authors never pictured. You cannot test the input you did not think of, and you did not think of it precisely because you wrote the code. For the longest time I tried to out-imagine my own code by hand, listing more and more inputs, and I kept losing, because the bug always lived one input past wherever my list stopped.

Property-based testing flips who supplies the inputs. Instead of writing examples, you state an invariant, a thing that must hold for every input, and let a generator like Hypothesis manufacture thousands of inputs whose only job is to break it. You assert the contract. The framework plays the adversary, and it is a far better adversary than I am, because it has no mental model to protect.

The central invariant for my router was simple to say, for any non-empty task, routing either returns a capability the registry actually holds, or it rejects the input with a typed error, nothing in between. State that, point a few hundred generated text inputs at it, and the framework goes hunting for the counterexample. The others have the same shape, history length never decreases across a run, the loop detector always returns a real boolean, the selected capability is always a string. Each one is a property paired with a strategy for generating inputs.

And it finds things. A single-character task that hits a branch I forgot existed. Unicode combining characters that quietly collapse the loop detection. Whitespace that strips to empty with no downstream guard. Inputs I would never, ever have typed, because typing them requires not understanding my own code, which is the one thing I cannot do on purpose. Pushing the example count to a few hundred per property costs a couple of seconds and catches far more than the default.

If you only test the inputs you can imagine, you only test the bugs you do not have. State the invariant and let the machine go looking for the ones you do.

Part of a series. Start here: A green test suite proves less than you think