AI code conversion fails on the system around the code, not the code

Converting old code with AI goes wrong less in the new code than in everything around it. Stephen Chin of Neo4j and Neeraj Deshmukh of Blitzy made this case on the AI Engineer show, in an episode published on 6 November 2026.

What was said

Their running example was an interest calculation written in COBOL, an old business programming language still used in banks and insurers. The output is money. The result depends on the job that runs it and on the account updates, not only on the formula.

Ask an AI to turn that formula into Java and it finds the formula and converts it. It does not find the record precision, the account updates, the transaction output or the batch contract, the rules that govern the scheduled jobs. A speaker on the show put it this way: "So the failure here is not that AI can't write the code. It's that AI doesn't know the 14 other places where that logic, that code is being used."

The speakers said bigger context windows do not fix this. A speaker said models advertise a million tokens, but usable context is 200,000 or less, and accuracy drops as it fills. Their answer was to map the code as a graph of relationships and hand each AI agent only the slice it needs.

They gave one small failure. A speaker said COBOL and Java round differently, so the new code compiled and looked right, yet it was off by one cent. The speaker said a parity test, which compares new output with old, caught it. Another line was blunt: "you have to prove the parity, you can't take it as an assumption or as a promise."

Why it matters

Our reading: the dangerous conversion is the one that looks finished. Code that compiles and returns plausible numbers passes a casual review. A speaker said a one-cent gap, repeated over a million transactions, becomes a real problem.

AI makes this faster because it can rewrite far more code than a team can check by hand. Anyone buying modernisation work should ask how the vendor proves the new system matches the old on real outputs. Judging only how well the AI writes code misses the point. A speaker said they spend more compute on testing than on code generation.

The other side

The evidence here comes from the speakers' own work. Chin works at Neo4j, which makes graph databases, and Deshmukh at Blitzy, which sells the approach. The results they described, from a bank and an insurer, came with no outside check in the excerpts.

The excerpts also leave open what the graph can miss. A map built from code may not capture every place logic is used. The talk gave no figures on how often parity tests fail or how much human review the process still needs.