OpenAI Claims Breakthrough in Legacy Language Migration with 97.9% COBOL Accuracy

OpenAI released a research report on October 20 announcing enhanced capabilities for its GPT-6 Astra and other models in building and migrating "low-resource programming languages" such as COBOL, Fortran, and Visual Basic. Although these languages have fewer public code samples and documentation, they widely support critical systems including banking, ATMs, weather forecasting, and nuclear safety analysis.

To measure this capability, OpenAI introduced two benchmarks: MultiLangBench, which requires models to build a SQLite-compatible database engine from scratch in 20 languages; and another benchmark requiring models to migrate COBOL programs to modern languages like C++, Java, Python, or Rust while preserving original business logic.

The results show that GPT-6 Astra achieved an average test pass rate of 87.4% across the 20 languages in MultiLangBench, a significant improvement over the previous generation GPT-5.6 Sol's 26.8%. Specifically, the COBOL pass rate rose from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. The lower-cost GPT-6 Luna achieved an average pass rate of 40.3%, with estimated API costs reduced by approximately 78% compared to GPT-5.6 Sol.

In the more challenging COBOL migration benchmark, GPT-6.1 Sol reached an accuracy of 86.6% at low reasoning intensity, far exceeding GPT-5.6 Sol's 5.4%, with costs reduced by 39%. At higher reasoning intensity, accuracy reached 97.9%, approaching the evaluation ceiling.

OpenAI stated that this progress helps enterprises maintain legacy systems, fix defects, and advance software modernization. However, the company also emphasized that using AI migration results in production environments still requires expert review and testing against specific business rules and dependencies.