読書リサーチレーダー投資フレームワーク
ログイン / 新規登録
← 読解に戻る
ニューストピック · 说服性AI认知防火墙重大5 分で読めるDisconfirmAI 編集部

Will Duncan: Persuasive AI Risk Is Underrated, Cheap LLMs Make Cognition Firewalls Now Tractable

LessWrong author Will Duncan argues that AI's persuasive capability is rising while the risk remains underrated and under-addressed, attributing community neglect to structural weaknesses in human and AI safety attitudes; psychological harm is hysteretic and costly to reverse; and cheap large language models have flipped the asymmetry between technical labor availability and content semantic classification, making individual-side cognition firewalls now tractable. He has built an open-source prototype called Tolerance that filters social media feeds at roughly $0.05 per thousand posts.

2026-10-20 21:58 GMT+8

LessWrong author Will Duncan's core judgment is that persuasive AI as a risk is underrated and under-addressed, while the technical conditions to counter it have only just become available. His argument runs on three levels: the threat is rising, the community neglects it due to structural weaknesses in human and AI safety attitudes, and cheap large language models now make individual-side cognition firewalls tractable.

On the threat itself, Duncan notes that current AI persuasion is only 'modestly and contextually superhuman.' He draws a distinction between lab results (Hackenburg et al., 2026) and real-world observation (Linch, 2026): models outperform professional persuaders in controlled settings, but there is little evidence of this in the wild. He considers this consistent with the current point on the capability curve, and adds that even if widespread AI persuasion were already happening, we lack the means to detect it.

Duncan compares persuasive AI to coding and mathematics. Coding has 'practically fallen,' mathematics is in the process of falling, and Anthropic's Jack Clark has put the recursive self-improvement timeline at 2027–2028. Against this backdrop, he argues the AI safety community treats persuasion as a subordinate problem due to 'structural weaknesses in human and AI safety attitudes.'

ログイン / 新規登録
読書リサーチレーダー投資フレームワーク

He draws a key distinction: AI has a stake in solving the current crisis in computer security because it relies on networked silicon to grow and improve. The psychological commons, however, are not necessary in the same way, so proactive efforts will be required to stabilize it.

Duncan uses 'hysteresis' to describe the character of psychological harm—it is difficult to heal and self-reinforcing. He cites depression recurrence statistics (50% of those who recover from a major depressive episode have at least one more in their lifetime), confirmation bias, in-group bias, and PTSD as evidence. This characteristic means the cost of reactive remediation far exceeds that of preemptive defense, making preventive deployment warranted.

Why was this not feasible before? Duncan identifies the technical precondition: in the past, writing customized social media filtering interfaces for each user required prohibitively expensive engineering labor. Now, 'as a result of cheap machine intelligence, the asymmetry in technical labor availability and semantic classification of content has flipped in favor of users.' This is his core argument for why cognition firewalls are 'now tractable.'

He built an open-source prototype called Tolerance. The tool targets the harm from current recommender systems—those that 'steal more time, attention, and sensemaking than they return in enrichment and socialization.' Duncan reports its cost at roughly $0.05 per 1,000 social media posts, amounting to a few dollars a month under heavy use. It can also hide streaming output until a model completion is finished (currently only on claude.ai), which he personally finds interrupts the usage loop and lessens compulsive use.

But Duncan maintains honest boundaries about his prototype: 'It is remarkably effective at its current task of altering my information diet, but I don't have enough data to say whether it generalizes.' The tool's current construction—prompts, fast cheap models, and hard-coded platform detection—he acknowledges will have to give way to aggregate data ingestion and countermeasure strategy by more intelligent models, more user-customization, trained ensembles of classifiers, filtering of arbitrary sensory data, and information sharing in consensus protocols and data unions.

On why the community neglects this risk, Duncan cites three cognitive biases: the third-person effect (people reliably believe media influences others' thinking and not their own), the bias blind spot (people see biases in others' thinking and not their own), and choice blindness (people defend choices they never made without noticing the swap). He considers these biases particularly threatening to 'a movement that prides itself on taking reality on a rational basis.'

He also points to a specific industry decision as evidence: OpenAI removed persuasion from its Preparedness Framework on April 15, 2025, stating that 'Persuasion risks will be handled outside the Preparedness Framework, including via our Model Spec, restricting the use of our tools for political campaigning or lobbying, and our ongoing investigations into misuse of our products (including detecting and disrupting influence operations).' Duncan argues this strategy is both incomplete because jailbreaks remain effective, and does nothing to curb the threat from open-source models.

Duncan uses cybersecurity as an analogy for what the defense ecosystem should look like—attack and defense learning from each other in iterative cycles. But he explicitly acknowledges the analogy's limits: 'human psychology is not trivially patched on any timescale or level of effectiveness relevant to the threat of AI persuasion.' The speed and specificity of the threat will operate on superhuman timescales, so countermeasures must interpret cognitohazards at superhuman speed and capability.

The ideal end state he envisions is cognition firewalls becoming standard features in operating systems. But this faces difficulties he acknowledges: mobile operating systems currently effectively forbid users from having this power (through degraded and feature-incomplete mobile web app experiences, throttled accessibility services, and native app encapsulation); these systems are themselves an injection and attack surface, requiring model diversity and privacy; and the most fundamental question—'Who decides what is harmful and how is delicate and requires open deliberation, choice, and engineering while the clock is ticking.'

At the article's close, Duncan adds a self-qualifier: when he says 'AI' he frequently delineates it strictly from humanity, but he does not consider this a completely coherent view. The boundaries between the physical and digital worlds are progressively blurring, and the best solutions lie in the hybrid space. He asks readers to understand the danger as stemming from 'lack of proactive choices around how AI is developed and deployed,' rather than AI itself.

ソース

  • lesswrong.com↗
トピック

说服性AI认知防火墙

随着大语言模型在说服力上的提升,一种被称为“认知防火墙”的技术概念正从理论走向实践。该概念旨在通过技术手段拦截或净化具有操纵性的AI输出,以保护用户的心理自主权和系统安全。

4 ステップ0 件更新日 2026-10-21 09:53

経緯

  1. 2021-12-24Beth Barnes发表《AI说服的风险》,论证AI伴侣和助手可能成为高效的心理操控载体,提出需警惕高保真意识形态传播带来的社会分裂。lesswrong.com ↗
  2. 2026-03-24eBay研究人员发布论文,提出针对浏览器AI代理的“认知防火墙”,采用边缘-云混合架构防御间接提示注入,将攻击成功率降至1%以下。arxiv.org ↗
  3. 2026-07-01罗马第三大学等机构发表论文,定义LLM安全的“认知防火墙”为主动运行时监督框架,通过意图、零信任上下文、一致性和输出风险四个门控来阻断多轮次越狱攻击。arxiv.org ↗

未解決の問題

  • 用户端认知防火墙(如Tolerance)在实际部署中如何平衡过滤效率与误杀率?
  • 针对LLM内部安全的“认知防火墙”架构能否有效防御具备自适应能力的红队攻击?
  • 当AI代理拥有执行权限时,现有的边缘-云混合防御机制如何应对更复杂的间接提示注入变种?
トピック全文を読む →
リサーチを続ける →