OpenAI Releases MultiLangBench to Evaluate AI Coding in Low-Resource Languages like COBOL

OpenAI has recently released MultiLangBench, a programming benchmark designed to test whether AI models can build SQLite-compatible database engines from scratch using 20 different programming languages. The benchmark specifically covers "low-resource" languages with limited training data and documentation, such as COBOL, Fortran, and Visual Basic, alongside mainstream languages like Python, Java, and Rust, allowing developers to compare model capabilities across diverse software development environments.

This initiative addresses a persistent challenge in enterprise IT: many core business systems have been running for decades and still rely on early-generation languages like COBOL. When companies need to maintain or rewrite these legacy systems, they often face issues such as the departure of original developers, incomplete documentation, and business rules embedded solely within the code. While AI can assist in program maintenance and cross-language porting, it must accurately understand existing business logic to avoid altering original calculation results or data processing methods during the rewrite.

MultiLangBench uses SQLite 3.40.1 as the common test subject, requiring models to write programs that handle database operations in specified languages based on provided documentation. The testing environment prohibits internet access and removes existing SQLite software to prevent models from simply calling pre-existing libraries. Results are verified against the original SQLite implementation to ensure functional correctness.