- OpenAI says GPT-6.1 Sol reaches 86.6% accuracy on its own COBOL migration evaluation at low reasoning effort, up from 5.4%. A migration counts only if every hidden test passes.
- A rewrite can run without errors and still give different financial results, so the useful test is whether outputs match, not whether the system starts.
- Before a pilot, record your system's real inputs and outputs, and have experts review every difference between old and new.
A migration that runs is not a migration that is right
The hard part of moving old software is not getting the new code to run. It is getting the new code to give the same answers. OpenAI's latest announcement on COBOL, Fortran and Visual Basic is a useful reminder. The way it tests migration sets a standard that buyers can reuse.
OpenAI published its results on October 19. They are the company's own evaluations, run on its own models. The source we reviewed gives no sign that anyone outside OpenAI has reproduced them.
What OpenAI reported
OpenAI says GPT-6.1 Sol reaches 86.6% accuracy on its COBOL migration evaluation at low reasoning effort. The earlier GPT-5.6 Sol scored 5.4% at the same setting. OpenAI says the new model costs 39% less.
At higher reasoning effort, OpenAI says accuracy reaches 97.9%. The company says that brings the evaluation close to saturation. That means little room is left to measure further gains.
MultiLangBench is the second test. A model must build a database that behaves like SQLite, and it does so in each of 20 programming languages. This test reports a test pass rate, which counts individual tests. At maximum reasoning effort, OpenAI reports GPT-6 Astra averaging 87.4%. GPT-5.6 Sol managed 26.8%.
For COBOL on that test, the pass rate rose from 10.9% to 72.1%. OpenAI says COBOL stayed the hardest language of the 20.
How the migration test works
In the migration test, models convert COBOL programs into C++, Java, Kotlin, Python or Rust. Hidden tests then check the results. OpenAI says the outputs must match exactly, down to each byte. A migration counts as a success only if every test passes. The 86.6% is therefore an accuracy score across whole programs, not a share of individual checks.
That strictness matters because of how COBOL handles numbers. OpenAI explains that its fixed-point arithmetic drops extra decimal digits unless rounding is requested. A rewrite that rounds instead can run without errors. It can still produce different financial results.
A demo may not reveal this kind of failure. Imagine a rewrite where the new system starts and the screens load, but the totals differ from the old ones. The lesson here is that the useful question about any migration is not "does it work?" but "does it match?"
Why the stakes are high
OpenAI points to how much rides on COBOL. It says the language sits behind more than 40% of online banking systems. By its count, about four in five card payments made in person and 19 in 20 ATM transactions also depend on it. OpenAI adds that Fortran appears in weather forecasting and in safety analysis for nuclear reactors.
OpenAI also describes the people problem. Documentation may be incomplete. The original developers may have left. Some business rules may exist only in the code.
That last point is the real risk. In some such systems, the code is the only record of what the business promised to do. A migration that changes behaviour quietly rewrites those promises. The cost lands downstream. It falls on the customer whose balance is wrong, or the finance team that must trace why.
What the numbers do not show
Several limits apply. The scores come from the vendor's own tests. OpenAI says it is releasing the MultiLangBench setup and results on GitHub, so developers can inspect them. The source does not say the same for the COBOL migration evaluation.
The 86.6% figure is for low reasoning effort. The source does not describe the programs in the hidden tests. We cannot tell how closely they resemble a bank's production code.
OpenAI adds its own caution. Applying these abilities to production software still needs expert review. It also needs testing against the system's business rules and dependencies.
What leaders should ask
If your organisation runs COBOL, Fortran or Visual Basic systems, these results justify a pilot. They do not justify skipping checks. Put four questions to your team.
First, do we have recorded inputs and outputs from the running system to compare a rewrite against? Second, does the comparison cover rounding, truncation and missing records, not just the common path? Third, who signs off on a difference, and who understands the rule behind it?
Fourth, is the extra accuracy at higher effort worth the extra cost? OpenAI reports 86.6% at low effort and 97.9% at higher effort. It gives no cost for the higher setting in this announcement. Ask for that figure before you plan a pilot budget.
The tests that decide a migration are the ones you write from your own system's behaviour. AI may now be able to write the new code. Only your organisation can say what the old code was meant to do.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: OpenAI.
