LesenForschungRadarAnlageframework
Anmelden / Registrieren
← Zurück zur Lektüre
NewsThema · MKA基准测试Wesentlich2 Min. LesezeitDisconfirmAI Redaktion

Google DeepMind and Artificial Analysis Launch Multimodal Knowledge Agent Benchmark to Evaluate AI on Complex Visual Workflows

The new MKA benchmark focuses on visual knowledge work involving images, charts, and layouts, filling a gap left by text-only evaluations.

2026-11-11 00:00 GMT+8

Google's DeepMind team, in partnership with international AI evaluation firm Artificial Analysis, launched the "Multimodal Knowledge Agent" (MKA) benchmark on November 10, 2026. This new standard aims to evaluate how well AI agents perform real-world knowledge work tasks that require reasoning across visually complex files.

Unlike previous benchmarks that relied primarily on text-only inputs, MKA targets the visual nature of everyday professional work, such as cross-referencing architectural blueprints, tracing engineering schematics, and synthesizing findings into polished decks or reports. To ground the benchmark in reality, Google collaborated with domain experts across 10 verticals and 83 occupations to create a 200-task evaluation set. Each task simulates a multi-step project that takes a seasoned professional three to eight hours to complete, paired with an expert-written rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design.

To ensure integrity, Artificial Analysis runs the 200-task evaluation set privately, preventing task leakage into training data or discovery via search. A separate set of 10 public tasks has been released on Hugging Face for developers to inspect the format and test their own agents. All models run within the same open-source agent harness, Stirrup, using identical tools, sandboxes, and turn budgets.

Google claims that its latest model, Gemini 4 Carbon, ranks second in quality among tested models on this benchmark, while completing tasks at six times lower average cost compared to the top-scoring model. These results place Gemini 4 Carbon on the Pareto frontier of quality and price for MKA evaluations. However, these performance metrics are currently based on vendor self-testing and have not yet been independently reproduced by third parties.

Anmelden / Registrieren
LesenForschungRadarAnlageframework

Quellen

  • blog.googleGoogle↗
Thema

MKA基准测试

谷歌DeepMind与评测机构Artificial Analysis联合推出多模态知识代理基准测试(MKA),旨在评估AI智能体处理视觉复杂文件并执行现实世界专业任务的能力。该基准填补了现有评测仅依赖文本的空白,要求模型从建筑蓝图、工程示意…

3 Schritte1 BeiträgeAktualisiert 2026-11-14 13:20

Wie es dazu kam

  1. 2026-11-10谷歌DeepMind与Artificial Analysis正式宣布推出MKA基准测试,并发布技术报告及10项公开示例任务。blog.google ↗
  2. 2026-11-10Artificial Analysis发布MKA排行榜,显示Claude Fable 5.2 (57%) 领先 Gemini 4 Carbon (54%) 和 Claude Opus 5.5 (53%)。artificialanalysis.ai ↗
  3. 2026-11-11IT之家等媒体跟进报道,强调Gemini 4 Carbon在成本效益上的优势及其在特定垂直领域的表现差异。ithome.com ↗

Was noch ungeklärt ist

  • MKA私有测试集是否会定期更新以防止模型过拟合?
  • 各模型在不同行业垂直领域(如医疗 vs 法律)的性能差异是否会导致专用智能体的兴起?
  • 移除视觉工具后模型性能下降的具体幅度是否足以证明当前OCR替代方案的局限性?
Vollständiges Thema lesen →
Recherche fortsetzen →