I spent the early part of my career believing that if I just worked hard enough — patched fast enough, locked down enough ports, wrote enough policy — I could build a system that wouldn't fail. Somewhere along the way, I stopped believing that. Not because I got cynical, but because I started paying attention to how complex systems actually fail.
There is no such thing as a perfectly secure system. There isn't even such a thing as a perfectly <em>safe</em> system, and industries far older than software have the scars to prove it. Aviation, arguably the most safety-obsessed engineering discipline on the planet, has stories of aircraft designed with safety in mind at almost every level — undone by scenarios so bizarre that no reasonable designer would have dreamed them up. Power grids, hardened against storms and sabotage, get taken down with astonishing regularity by squirrels doing perfectly ordinary squirrel things. Reality is stranger than fiction, and it does not consult your threat model.
Once you accept that, the whole framing of security changes. The question stops being <em>"How do I prevent every bad thing?"</em> and becomes <em>"When something goes wrong — and it will — how quickly can we recover, how small can we keep the blast radius, and how do we evolve to meet the moment?"</em> That's resilience. Prevention is one tool inside it, not the goal itself.
The illusion of control is the real vulnerability
Here's a pattern I've learned to treat as a red flag: a team that feels <em>in control</em>. When a security or platform team tells me they have full control over their delivery lifecycle, my honest reaction is — are you sure? Humans aren't deterministic. Software, despite our fondest wishes, isn't fully deterministic either. Verifying that a program does exactly what its designer intended remains one of the hardest problems in computer science. A team that believes it has everything locked down has usually just stopped looking at the places where it doesn't.
The tell is where the money goes. Teams chasing a sense of control invest in things that make them <em>feel</em> good — more gates, more dashboards, more sign-offs. Teams building resilience invest in things that <em>minimize impact</em> — redundancy, graceful degradation, fast recovery, practiced incident response. Those are very different budgets, and only one of them holds up on a bad day.
There's an old ops joke that captures it perfectly: backups always succeed; it's restores that fail. Everything looks resilient until the moment you actually need it to be.
Software's unfair advantage — that we barely use
Here's the part that frustrates me most. Software has a luxury that no physical discipline enjoys: we can clone reality and break the clone.
A city planner cannot flood a neighborhood to find out which drains back up first. There is no ethical way to run that experiment. But I <em>can</em> stand up a faithful replica of a production system and hurt it on purpose — kill a node, sever a dependency, revoke a credential mid-request — and watch exactly what happens, at low cost and zero human risk. Other engineering disciplines would do unspeakable things for this capability. We have it, and most of us leave it on the shelf.
Yes, redundancy and simulation cost money. Running polyglot systems across multiple providers is genuinely expensive, and every organization has to decide how badly it wants uptime. But a lot of resilience work isn't expensive at all — it's just unfamiliar. You don't need to start by unplugging things in production.
Start embarrassingly small
The experiments I recommend to teams are almost anticlimactic. Take one assumption you'd bet money on — <em>"our login page always requires this cookie"</em>, <em>"this header is always validated"</em> — and actually test it. Duplicate a request, strip the thing you assume is essential, and see what happens. The blast radius is tiny. The findings, in my experience, are frequently not.
What you're really testing isn't the system. It's your mental model of the system. And the gap between the two is where incidents live.
The other thing I push for: don't run these experiments alone. Security teams are oddly hesitant to walk across the aisle to platform engineering and say, "Want to co-conspire on this?" Yet the assumptions most worth testing are the <em>jointly held</em> ones — the things both teams believe, that neither team has poked in years.
The earthquake test
A geologist named Susan Elizabeth Howe put it in a line I think about constantly: a building doesn't care whether the earthquake was predicted or not — it either stays up or it doesn't.
That's the whole philosophy in one sentence. Our industry pours enormous energy into prediction: threat intel, risk registers, likelihood scores. Prediction is fine. But when the ground actually moves, none of your forecasts are load-bearing. The only thing that matters is what you built.
So that's the question I ask about every system I'm responsible for now. Not "have we predicted the failure modes?" — we haven't; nobody ever has. Just: when the shaking starts, does this thing stay up? And if it falls, how fast can we rebuild?
Design for that, and prevention takes care of the rest far better than prevention alone ever did.
