Microsoft Open-Sources Scope to Evaluate and Trace AI Coding Agents
Microsoft has open-sourced Scope, an evaluation framework for AI coding agents like GitHub Copilot and Claude Code. It helps development teams verify correct SDK and API usage by logging execution steps and assessing results, addressing reliability issues where code works but doesn't follow intended workflows.
On October 19, 2026, Microsoft open-sourced Scope, an evaluation platform designed to help development teams test and track the performance of AI coding agents such as GitHub Copilot and Claude Code when executing specified tasks. The platform allows software teams to verify whether agents correctly use Software Development Kits (SDKs), Application Programming Interfaces (APIs), and related tools according to documentation, while recording the entire operation process and test results.
In practice, AI agents assisting developers can make mistakes such as selecting the wrong API, ignoring important instructions in documentation, or failing to complete work after encountering errors. Even if the final generated code runs successfully, it does not guarantee that the agent used the expected toolchain. Scope addresses this pain point by enabling teams to test error-prone aspects of agent interactions with proprietary SDKs, command-line tools, and Model Context Protocol (MCP) servers used to connect external services.
Development teams can define completion criteria for tasks and provide starting code, allowing different AI agents to execute the same task. The system records the AI model version, agent version, and tool configuration for each run. After modifying documentation or tools, teams can re-test with the same task to observe if agents complete work more easily and distinguish between sporadic and recurring errors. Additionally, Scope supports setting multiple check conditions with dependencies (e.g., successful build before checking SDK initialization) and uses large language models to assess condition fulfillment, preserving operation logs and directory snapshots for troubleshooting.