A retest is better evidence than a closed ticket, says Horizon3.ai

The scoreboard rewards the wrong thing

Most security reports count finished work. Tickets closed, backlog down, severity scores lower. Horizon3.ai, a security testing vendor, argues in an October 29 post that these counts say little about the real question. What could an attacker do before the fix, and what can they do now?

The post draws on three customer cases. Horizon3.ai does not name the customers. The results come from its own NodeZero testing product. Treat them as illustrations of an argument, not proof of a wider trend.

A ticket can hide the attack path

Remediation queues break work into single findings. Each gets a severity, an owner and a deadline. Horizon3.ai says this can cut a weakness off from the attack path that gives it weight.

Its first case is a large equipment rental company. An employee launched a test web server written in PHP and ran it from the root directory. A web server shares the files in the folder it runs from. Run from the root directory, it can share far more than intended. Here, that left SSH key pairs exposed. These are credentials that let one machine log into another. Horizon3.ai says this opened a chance for lateral movement, which means hopping from machine to machine. It says NodeZero flagged the problem about eight hours after the change.

Deleting the exposed files would not end the risk, the post says. Someone may already have working keys. The post leaves open which systems accepted the keys and what they allowed. The real risk depends on those answers.

Now suppose security finds the flaw but another team must fix it. Horizon3.ai says that team needs three things: the route an attacker could take, the proof, and fix guidance. A line in a queue does not supply them. This case has no retest result in the post. It shows the path problem only.

Why small flaws can add up

This is our argument, not a finding from the post. Attackers often link modest weaknesses rather than rely on one large hole. Here is a hypothetical example. An exposed key opens a system that trusts it, and that system holds an account with wide permissions. A severity score rates each piece alone. A tested attack path rates the whole route.

Fewer findings is not the same as less exposure

The second case is a global investment firm. Horizon3.ai says a retest of the same scope still found 46 minor weaknesses after fixes. Yet the impacts the test demonstrated fell from 251 to zero. It also reports compromised hosts down from 67 to zero and compromised credentials down from 52 to zero.

Picture two ways to brief leadership. "46 findings remain" sounds like failure. "Zero impacts" sounds like success. Horizon3.ai says leaders need both numbers, with context. A clean result only describes what happened within the test's "scope and conditions". Remaining weaknesses still need tracking. Threat intelligence and asset value still shape priorities.

The lesson here is simple. Backlog size measures effort. A retest measures outcome, inside the limits of that test. The two can move apart.

Someone must own the check

The third case is a healthcare software provider. Horizon3.ai says its first AWS test found 16 weaknesses tied to severe business harm. They included takeover of a full AWS account and exposure of sensitive data. After fixes, the retest found two low-severity weaknesses. It could not chain them into meaningful business harm.

The post's point is about process. Name who verifies the fix at the time you name who makes it, before the ticket changes hands. If the retest brings back the old result, the ticket returns with proof of what is still open. If the fix holds, the team has grounds to close it. Testing should continue after that. New identities, permissions, systems and connections can open another route to the same harm.

What to ask your team

Horizon3.ai suggests starting with one exposure on a business-critical system. Review it with the people who make the changes. Ask what route and impact the test has shown. Ask what must change and who owns each change. Ask what proof will show the fix worked.

Then check what leadership receives. Does the report show what an attacker could have done before the fix, and what changed after? Does it show whether the gain lasted? If not, change the process at the next exposure review. Keep the proof, name an owner for verification, and report the result.

A closed ticket records that work was done. A retest shows whether it changed what an attacker could do in the test.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Horizon3.ai.