Ask a security team what their "security coverage" is and you'll usually get a confident percentage. Ask what that percentage actually <em>means</em> and the conversation gets much quieter. Coverage of what? Against whom? Measured how? Nobody quite knows — someone defined it years ago, it goes up and to the right, and everyone has agreed not to look at it too closely.
I've started calling this whole genre of activity metrics theater, and the longer I work in and around security, the more convinced I am that it's not just useless — it's actively harmful. A misleading metric is worse than no metric, because it manufactures confidence exactly where scrutiny should be.
The vulnerability count trap
Take the most beloved security metric of all: the number of vulnerabilities discovered. Every dashboard has it. Every board deck has it. And it's almost perfectly designed to be misread.
Here's a scenario I've seen play out more than once. An organization invests in developer relationships, makes it easier and safer to report problems, and the number of reported security issues goes <em>up</em>. On the dashboard, this looks like things are getting worse. In reality, it's one of the healthiest signals you can get — engineering teams surfacing issues instead of burying them means trust is growing between teams that historically didn't talk. The number climbed <em>because</em> things were improving. Correlation and causation are not the same thing, but a red trend line on a dashboard doesn't come with that footnote.
Now flip it around. A platform team gets handed a list of a thousand vulnerabilities to remediate. How many of those live in components that aren't even exposed to the public internet? Should those be prioritized? Almost certainly not — but the metric says "a thousand open vulns," so the pressure lands anyway. And here's the dark punchline I've encountered in real organizations: while everyone grinds through that list, the credentials to the admin system that hooks into everything are sitting in a plain text file on someone's desktop. Attackers do a kind of math — call it attacker calculus — about effort versus payoff, and our dashboards almost never reflect it. We count what's countable, not what's exploitable.
Meanwhile, the numbers CISOs present up the chain are frequently not the ones boards and executives actually need. A board doesn't want a CVE count. It wants to know: if something breaks, what happens to the business, and how fast do we get back up? Very few dashboards even attempt to answer that.
When compliance actively makes you worse
Metrics theater has an older, more institutionalized sibling: compliance theater. And I want to be careful here, because the <em>intent</em> behind most regulation is good, and in year zero a fresh compliance regime often genuinely helps. The problem is what happens next. Regulation calcifies. The world moves; the checklist doesn't. What helped in year one can be quietly eroding your resilience by year three. There's even solid research showing that greater compliance doesn't reliably produce better security outcomes.
My favorite illustration: imagine a team that eliminates SSH access to production entirely. This is an unambiguous security win — standing SSH access is precisely the kind of door attackers love finding open. Now imagine the audit. The checklist, written in a different era, effectively <em>requires</em> SSH access to be present and managed. So the team that made itself safer has to spend years explaining to auditors why it can't tick a box that would make it less safe. When the checkbox becomes the thing standing between you and a certificate, the checkbox wins — and the mission loses.
The general principle is one I've come to see everywhere in complex systems: fragility accumulates when practices outlive the world that justified them. The status quo moves on; the process doesn't; the gap between them is where resilience quietly drains away. Nobody decided to become fragile. They just kept doing what the checklist said.
"We patched it" is not the same as "we're fine"
One more piece of theater, this one beloved by incident retros: the vulnerability gets patched, the ticket gets closed, and everyone declares the matter resolved. If that vulnerability actually resulted in an outage or a breach, it is <em>not</em> resolved. You've patched one door that one attacker happened to use. The underlying questions — why was the blast radius that big, why did detection take that long, what does this incident say about our mental model of the system — remain wide open. Closing the ticket updates the dashboard. It doesn't update your understanding, and the understanding is the part that was actually broken.
The only defense: people who ask why
I don't have a magic replacement dashboard to sell you. What I can tell you is what the resilient organizations I've seen have in common, and it isn't better charts. It's people — often annoying people, bless them — who look at any metric, any checkbox, any long-standing practice and ask: <em>why? Why is this here? What decision does this number actually inform? What would have to be true for this green light to be lying?</em>
Every metric on your dashboard should have to survive that interrogation. Most won't. The few that do are the ones worth keeping — and they're almost never the ones about coverage percentages or raw vulnerability counts. They're the ones that tell you, when the bad day comes, whether the business stays standing and how fast it gets back up.
Measure that. Everything else is set dressing.
