My fallback became the only path anyone took

If you’ve ever added a fallback so a rare failure wouldn’t hurt anyone, go and check what fraction of your traffic is taking it. Mine was taking all of it.

The code was reasonable. A verification widget sits in front of my sign-up. Widgets can be slow, blocked by an ad blocker, or unreachable. So I wrote: wait six seconds, and if the widget hasn’t answered by then, let the visitor through anyway.

The comment above it says the point is that a slow or blocked widget should never strand a new visitor at the top of the funnel.

The widget answers in eight and a half seconds.

A timer doesn’t know it lost

The plan assumed the fallback would fire occasionally, for the unlucky few. Nothing in the code checks whether that assumption holds. A timer fires when it fires. It doesn’t report that it beat the thing it was covering for.

So the system worked exactly as written and produced the opposite of what it was written for. Every visitor got the emergency path, and the server refused every one of them for arriving without the widget’s answer.

Zero new users for thirty-five days.

None of them knew, and neither did I, because I’d built the emergency path to be silent. The silence was deliberate. I didn’t want to alarm anyone over a rare hiccup.

A rare hiccup that happens every time stops being a hiccup and becomes the product.

Why it took five weeks to see

Because I had a check, and my check could only ever say yes.

I knew the widget was involved, so I wrote one: load the page, count the widget’s frames. Zero frames means the widget never rendered. It came back zero every time. Twenty readings on one morning, all zero.

Zero was what success looked like. The widget renders no frame when it decides a visitor is fine, which is the whole design. A healthy widget and a dead one produce the same reading, and I was using that reading to tell them apart.

I found out by opening the page myself and looking at the field the widget actually writes to.

Full.

A valid answer, on a widget I’d spent three weeks describing as dead.

Nobody asked the boring question underneath it: if this thing were working perfectly, what number would my check return?

The answer was zero. Knowing that on day one would have sent me looking somewhere else.

Timers are the smallest version of this

Every founder has a fallback like it. The pricing exception you’ll make if the deal is good enough. The manual workaround for when the automation fails. The plan B you agreed on so plan A wouldn’t carry all the risk.

Each one is built for the rare case. Each one is easier than the thing it backs up. And none of them tells you how often it fires.

What I’d do instead

Count it.

Whatever your fallback is, make it say so when it fires. A line in a log, a tally, a note at the end of the week. If it’s firing every time, you’ll know in days rather than weeks.

Then, before you trust any check you’ve built, write down what it returns when the system is healthy.

If that value is the same as the one it returns when the system is broken, you don’t have a check. You have a number that will agree with you.

I’ve set the timer to fifteen seconds for now, which buys about six seconds of headroom on a good connection.

That’s a patch, and I’ve written down that it’s a patch, because the version of me in November will otherwise read it as a decision.