An AI total is only worth signing if each figure traces to a page or invoice line

On the AI Engineer podcast, Natalia Connolly and Kevin Madura of AlixPartners described a procurement review in which every answer had to cite a contract page or an invoice line. Their first Claude Code results looked right but held errors.

What was said

The guests pointed Claude Code at a post-merger review. A health company, called Harbor Health in the talk, had bought a smaller plan. Both used one vendor, Acme Health Services. The procurement lead would face Acme across the table, so the team needed a source for every point.

The first output looked polished. A speaker on the show said: "They look complete. They're certainly well formatted. They look plausible enough, you know." The team then checked every answer. Yearly spend, the division between the two firms and the number of contracts held up. Total contract value was off by about $800,000. The agent also gave December 1 as the last day to stop an auto-renewal. The real date was November 1.

The missing money traced to an amendment that was not linked to its parent contract, so the agent never found it. The guests named other traps. When several amendments each restate the total value, adding up the extracted dollar amounts overstates it. A very long contract can also stall in processing, with no sign it never reached the database.

Their answer was a five-stage pipeline. It scanned documents, sorted each by type, pulled out terms, rates and spend, and mapped services to spend categories. Only then did Claude Code analyse the results. A speaker said the team chose to "enforce some of that business logic in code versus just relying on a prompt."

Why it matters

Our reading: whoever signs off on an AI analysis carries the cost when it is wrong, and a polished answer hides where it can fail. Here that meant a total off by about $800,000 at a negotiating table, and a renewal that would have rolled over.

The guests structured the data first and enforced rules in code. That also let the team check every answer programmatically. A citation shows what the agent used. It cannot show what the agent missed, such as an unlinked amendment or a contract that failed to process. So a buyer should ask for the trail and for a list of documents that failed or stayed unlinked.

The other side

Traceability helps only if someone uses it. The AlixPartners team caught the errors because they checked every answer themselves. The excerpts do not show what happens when a source page is itself wrong.

The guests also called letting Claude work on unstructured data a valid approach. Structuring first suited this job because they knew up front which items they wanted, such as rate tiers and rebates. Madura said some of this handling may not matter in 6 to 12 months, adding "We'll see."