Replit Releases ViBench 2.0: Top AI Models Pass Only 49-65% of Full Product Builds

Replit released ViBench 2.0 on October 28, 2026, a new benchmark designed to evaluate AI coding agents on "long-horizon software development." Unlike most coding benchmarks that provide well-defined tasks within an existing codebase, ViBench 2.0 requires agents to start with an empty project and build a whole product one feature at a time, modeled after applications like Uber, GitHub, or Google Docs.

The benchmark consists of 17 products, 241 feature requests, and 544 test plans. It tests complex scenarios such as data consistency, concurrency, and state management across interacting features. For instance, in a code-hosting simulation, the agent must ensure that an editor opened before someone else saved does not erase their newer changes.

According to results published by Replit and its partner Fireworks, the best-performing models—GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5—pass all test plans in only 49% to 65% of their builds. This means that in more than a third of attempts, the resulting product contains at least one critical failure. GPT-6.1 Sol achieved a 65% pass rate at a cost of about $12 per build, while Opus 5.5 had a 49% pass rate but completed builds faster, averaging 70 minutes.

Failures were often due to subtle details rather than core logic errors, such as mishandled edge cases, database initialization issues, or conflicts at feature intersections. Nearly three-quarters of failed builds broke specifically where features interacted. These findings indicate that while AI has made significant strides in assisted coding, it still faces substantial reliability challenges in autonomously building and maintaining complex software products.