- GPT-6.1 Sol migrated COBOL to C++, Java, Kotlin, Python or Rust with 86.6% accuracy at low reasoning effort and 97.9% at higher effort, against 5.4% for GPT-5.6 Sol.
- OpenAI's test demands byte-for-byte output parity, which targets the behavior-preservation problem IBM said in February that code translation “captures almost none of.”
- OpenAI's evaluation covers program logic and outputs, while transaction systems, batch scheduling and mainframe data stores remain untested and still sit with the bank's own engineers.
OpenAI says its GPT-6.1 Sol model can now migrate COBOL programs into modern languages while preserving their exact behavior 97.9% of the time, up from 5.4% for the previous generation, and that result shifts the hardest problem in bank and government software modernization from writing new code to proving the new code behaves exactly like the old. The figures come from an OpenAI research post published on October 19, built on an internal evaluation that only counts a migration as successful when every hidden test matches the original program's output byte for byte.
GPT-6.1 Sol passes 97.9% of COBOL migrations with every hidden test intact
OpenAI built two new evaluations for what it calls low-resource programming languages, the older languages with too little public code for models to learn from easily. The COBOL migration test asks a model to rewrite existing COBOL programs in one of five modern languages and then runs hidden tests against the result. A single mismatched output fails the whole migration, so the score measures whether the rewritten program still does exactly what the original did.
The jump also came with a lower bill. At low reasoning effort GPT-6.1 Sol reached 86.6% while costing 61% of what GPT-5.6 Sol cost on the same run, and more thinking time took it to 97.9%, a level OpenAI itself describes as close to saturating the evaluation. In practice, a reasoning model that once failed nineteen of every twenty migrations now fails about one in fifty.
OpenAI's test targets the behavior gap IBM said translation misses
The evaluation reads as a direct answer to the argument IBM made in February, when a rival AI lab's post on COBOL modernization sent IBM shares down 13% in a single session. IBM's software chief responded that converting code between languages leaves the real work untouched.
“Translating code is one thing. Modernizing a platform is something else entirely.”
Rob Thomas, SVP IBM Software, “Lost in Translation”, IBM Newsroom, February 23, 2026
OpenAI's design accepts half of that critique. Its post gives a concrete example of why naive translation breaks financial software: COBOL's fixed-point arithmetic discards extra decimal digits unless rounding is explicitly requested, so a rewrite that rounds instead runs cleanly while producing different money totals. Byte-for-byte parity on hidden tests is built to catch exactly that class of silent error, which is the behavior IBM argued translation misses.
| ATM transactions | 95% supported by COBOL |
| In-person credit card transactions | 80% |
| Online banking systems | More than 40% |
| Hardest language on MultiLangBench | COBOL, 72.1% for GPT-6 Astra |
The other half of IBM's argument survives untouched. The evaluation measures programs and their observable outputs, and the post makes no mention of the transaction monitors, batch job control or mainframe data stores that surround those programs in a real bank.
| Modernization layer | Tested by OpenAI's COBOL evaluation? |
|---|---|
| Program business logic and outputs | Yes, byte-for-byte on hidden tests |
| COBOL language quirks such as truncation and rounding | Yes, called out explicitly |
| Transaction processing and online systems | Not described in the evaluation |
| Batch scheduling and job control | Not described in the evaluation |
| Data architecture and mainframe data stores | Not described in the evaluation |
| Production rollout and regulatory sign-off | Outside scope; OpenAI says expert review is still required |
Source: OpenAI, October 19, 2026; IBM Newsroom, February 23, 2026. Assessment: Santage.
The bottleneck moves from writing code to owning the test suite
When a model can preserve behavior in 49 of 50 programs, the scarce asset in a migration becomes the tests that define correct behavior. OpenAI's score depends on hidden tests that it wrote, and a bank's equivalent is decades of business rules that often exist only in the code itself, as OpenAI acknowledges when it notes that original developers may have left and documentation may be incomplete. A migration team with a thin test suite will get a confident rewrite that passes the tests it has and fails on the cases nobody wrote down.
At 97.9%, the remaining 2.1% is where the money is: the edge cases that no test captured are the ones a bank discovers in production, so the value in COBOL modernization now sits with whoever can prove two systems behave identically.
The broader benchmark points the same way. On MultiLangBench, where models build a SQLite-compatible database engine from scratch in 20 languages, GPT-6 Astra lifted its average pass rate from 26.8% to 87.4%, yet COBOL stayed the hardest language at 72.1%. That gap mirrors what Santage found in enterprise testing last month, when top models fixed under 40% of real enterprise code, and it is a reminder that every result here is self-reported on OpenAI's own AI benchmark.
IBM's results on October 21 are the first test of the new argument
IBM reports third-quarter results on October 21, two days after OpenAI's post, and its mainframe and software businesses depend on the view that moving off the platform is slow and risky. Banks now have a vendor offering high, measurable behavior preservation at falling cost, which strengthens their hand in pricing talks even if they never migrate a line. OpenAI also published the MultiLangBench test suite on GitHub, inviting outside checks of its claims, while the COBOL migration set remains internal.
For the institutions that run on COBOL, OpenAI's numbers suggest the translation step is becoming cheap and reliable on well-tested code, and the work that remains is the work they have avoided for decades: writing down what their systems are supposed to do. The banks that turn their legacy behavior into executable tests will be able to buy migration as a commodity, and the ones that cannot will keep paying for the mainframe, whatever the models score.
Santage is committed to independent, transparent journalism. This article is produced in accordance with Santage's Editorial Standards and aims to provide accurate and timely information. Readers are encouraged to verify information independently.