読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · OpenAI多语编程基准重大2 分で読めるDisconfirmAI 編集部

OpenAI Releases MultiLangBench to Evaluate AI Coding in Low-Resource Languages like COBOL

OpenAI introduces a new benchmark testing large language models' ability to build database engines and port legacy code in data-scarce languages such as COBOL and Fortran.

2026-10-20 19:41 GMT+8

OpenAI has recently released MultiLangBench, a programming benchmark designed to test whether AI models can build SQLite-compatible database engines from scratch using 20 different programming languages. The benchmark specifically covers "low-resource" languages with limited training data and documentation, such as COBOL, Fortran, and Visual Basic, alongside mainstream languages like Python, Java, and Rust, allowing developers to compare model capabilities across diverse software development environments.

This initiative addresses a persistent challenge in enterprise IT: many core business systems have been running for decades and still rely on early-generation languages like COBOL. When companies need to maintain or rewrite these legacy systems, they often face issues such as the departure of original developers, incomplete documentation, and business rules embedded solely within the code. While AI can assist in program maintenance and cross-language porting, it must accurately understand existing business logic to avoid altering original calculation results or data processing methods during the rewrite.

MultiLangBench uses SQLite 3.40.1 as the common test subject, requiring models to write programs that handle database operations in specified languages based on provided documentation. The testing environment prohibits internet access and removes existing SQLite software to prevent models from simply calling pre-existing libraries. Results are verified against the original SQLite implementation to ensure functional correctness.

According to reports by iThome, OpenAI evaluated models including GPT-6 Astra. At maximum reasoning intensity, Astra achieved an average pass rate of 87.4% across the 20 languages, significantly higher than GPT-5.6 Sol's 26.8%. COBOL remained the lowest-performing language for Astra, with a pass rate of 72.1%. Additionally, OpenAI designed a COBOL rewriting evaluation for cross-language porting, requiring conversion to C++, Java, Kotlin, Python, or Rust while ensuring output consistency with the original program (e.g., matching rounding rules for monetary calculations). In this specific test, GPT-6.1 Sol achieved 86.6% accuracy at lower reasoning intensity. The benchmark configuration and test data are now publicly available on GitHub.

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

ソース

  • ithome.com.tw↗
トピック

OpenAI多语编程基准

OpenAI于2026年10月19日发布新基准测试MultiLangBench,专门评估大模型在COBOL、Fortran等“低资源”编程语言中从零构建SQLite兼容数据库引擎的能力,并公布GPT-6系列模型在遗留代码迁移任务上的显著性能…

3 ステップ2 件更新日 2026-10-21 14:52

経緯

  1. 2026-10-19OpenAI在GitHub开源MultiLangBench基准测试套件,要求模型在禁止联网和移除现有SQLite库的环境下,依据文档用指定语言实现SQLite 3.40.1功能。github.com ↗
  2. 2026-10-19OpenAI发布官方博客《Advancing coding in low-resource programming languages》,披露GPT-6 Astra在MultiLangBench上平均通过率87.4%,以及GPT-6.1 Sol在COBOL迁移评估中高达97.9%的准确率。openai.com ↗
  3. 2026-10-20多家科技媒体(如iThome、The Decoder)报道OpenAI的新基准,指出COBOL仍是最难的语言(Astra通过率72.1%),并强调数据为厂商自测,未经独立验证。ithome.com.tw ↗

未解決の問題

  • 第三方机构或开发者社区能否在相同条件下复现OpenAI报告的GPT-6系列高通过率数据?
  • 在实际企业遗留系统迁移项目中,AI生成的代码是否真能保持业务逻辑的字节级一致性,还是仅通过了基准测试的特定用例?
  • OpenAI是否会公开其未发布的COBOL迁移评估的具体测试集和评分标准以供外部审计?

このスレッド内の他の記事

  • OpenAI claims GPT-6 shows major accuracy gains in legacy programming languages like COBOL2026-10-21
トピック全文を読む →
リサーチを続ける →