LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · LLM智能体安全Material1 min de leituraEditorial DisconfirmAI

Preprint: Attention Tracking Outperforms Presence Detection in Separating Successful Prompt Injections, Reaching 0.949 AUROC on Qwen

New research compares five internal-state-based prompt injection detection methods, finding that signals reflecting response formation (like attention diversion) better identify successful attacks than those detecting mere injection presence.

2026-10-15 12:00 GMT+8

A study evaluating large language model internal states for prompt injection detection reveals that signals indicating "attack success" differ from those indicating "injection presence." Researchers assessed five distinct representation-reading methods on Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it models.

The findings show that methods trained to detect injection presence or attack-related concepts generally separate unsuccessful injections more strongly from benign inputs. In contrast, methods based on attention diversion, exposure, and role confusion separate successful injections more effectively. The "Attention Tracker" method provided the strongest and most consistent separation of successful injections, achieving an AUROC (Area Under the Receiver Operating Characteristic Curve) of 0.949 on Qwen and 0.887 on Gemma.

These results suggest that internal signals closer to response formation may better encode information about an attack's success. Detection performance also depends heavily on the prompt template. While the study data is public, as a preprint, its conclusions await independent reproduction.

Fontes

  • arxiv.org↗
Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento
Tópico

LLM智能体安全

LLM智能体正从对话工具转向具备执行权限的软件组件,其安全风险已从文本生成扩展至控制流劫持与状态污染。当前防御体系呈现“检测”与“隔离”两条路径:一方面,研究者发现仅靠外部文本监控难以捕捉隐性欺骗,转而利用模型内部状态(如注意力转移)进行白…

4 etapas1 peçasAtualizado 2026-10-16 20:39

Como chegamos aqui

  1. 2026-10-14预印本提出FIBS白盒探针,在SHADE-Arena基准测试中检测欺骗AUROC达98.3%,超越黑盒文本监控。arxiv.org ↗
  2. 2026-10-14另一项研究指出,关注响应形成的内部信号(如注意力追踪)比检测注入存在更能识别成功攻击,Qwen模型AUROC达0.949。arxiv.org ↗
  3. 2026-10-15Perplexity开源AI智能体沙箱SPACE,宣称在118次对抗测试中未发生逃逸,但该结果属厂商自测。siliconangle.com ↗

O que ainda está indefinido

  • SPACE沙箱的零逃逸记录能否通过第三方独立红队测试?
  • 白盒探针所需的模型内部访问权限在实际部署中如何平衡隐私与安全?
  • 针对“私下采纳、公开沉默”的多智能体攻击,是否有有效的跨代理一致性校验机制?

Também nesta thread

  • Preprint: Few Adversarial Agents Cause 60% of LLM Agents to Privately Accept False Answers, Public Endorsement Much Lower2026-10-16
Ler o tópico completo →
Continuar pesquisando →