Langfuse Engineers Demonstrate Full Evaluation Workflow for High-Stakes AI Agents: From Dataset Building to CI Auto-Blocking
Annabell Schäfer and Doneyli De Jesus from Langfuse hosted a hands-on workshop at the AI Engineer conference on evaluating high-stakes AI applications. Using a banking assistant as an example, they walked through the entire AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, choosing between LLM-as-judge and code evaluators, and implementing CI that automatically blocks prompt changes when scores drop. This video serves as a practical reference for AI engineers needing to establish rigorous evaluation systems.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
For AI engineers developing applications in high-risk domains like finance or healthcare, systematically evaluating model performance is often more challenging than training itself. Annabell Schäfer and Doneyli De Jesus from Langfuse hosted a hands-on workshop titled "Evals for High-Stakes AI Agents" during the AI Engineer conference, providing an end-to-end solution to this pain point.
The speakers used a banking assistant as a case study to deconstruct the complete AI engineering loop. The content covers foundational data tracing (including PII masking), building high-quality evaluation datasets, and translating domain expert feedback into specific failure modes. They also discussed evaluator selection, such as the use cases for LLM-as-judge versus code evaluators, and demonstrated how to implement automated monitoring via CI pipelines—where the system automatically blocks deployment if new prompts cause score drops.
Although no verbatim transcript was obtained, the video's chapter structure clearly outlines key steps in setting up an evaluation framework. For professional readers seeking actionable evaluation methods, particularly those focused on production safety, this is a resource with direct reference value.