Microsoft Principal Cloud Advocate Pamela Fox: Evaluating GPT-5.5 and Other Models via Scenario-Based Testing
At the AI Engineer World's Fair 2026, Microsoft Principal Cloud Advocate Pamela Fox demonstrated how to swap and test GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 on Microsoft Foundry. She covered scenarios ranging from letter counting to RAG groundedness, image input, and tool calling, introducing tools like ASSERT for generating test sets and DSPy for prompt optimization, emphasizing that real-world scenario testing reveals model differences better than generic benchmarks.
ImportanceMatérielPreuvesE2 non réplicableTraitementRapide
At the AI Engineer World's Fair 2026 in San Francisco, Pamela Fox, Principal Cloud Advocate at Microsoft, conducted a workshop on model swapping. She argued that generic benchmarks often fail to reflect how large language models actually behave in specific applications, urging developers to swap and test models directly within their own scenarios.
Fox compared GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 on the Microsoft Foundry platform. Her demonstration progressed from basic letter-counting prompts to complex tasks including Retrieval-Augmented Generation (RAG) groundedness, image input handling, multilingual pitfalls, tool calling, and building a travel agent with LangChain v1.
Beyond functional testing, she detailed evaluation methodologies such as string-check evals, LLM-as-a-judge, using Microsoft's new ASSERT tool to generate test sets from specifications, and optimizing prompts with DSPy to find instructions that even the cheapest model can satisfy. This sequence provided engineers with a practical workflow for model selection and evaluation.