LessWrong Author Will Duncan: Persuasive AI Is Underrated as a Risk; Cognition Firewalls Should Be Deployed Before Harm Becomes Widely Felt
LessWrong author Will Duncan argues that persuasive AI currently outperforms humans only in laboratory settings, but psychological damage is hysteretic — once it takes hold, it is difficult to reverse — so we cannot wait until harm becomes widely felt before acting. He contends that cheap LLMs have made content classification affordable on the user side, flipping the asymmetry in technical labor and making the construction of "cognition firewalls" a tractable engineering problem today. He developed a prototype called Tolerance as a proof of existence, while acknowledging that its generalizability has not been verified.
Will Duncan puts forward a specific judgment: persuasive AI is systematically underrated as a risk category, and the technical window to address it has opened. His core argument is not that "persuasive AI will destroy humanity," but a narrower yet more urgent proposition — because psychological damage is hysteretic, deploying defenses only after harm becomes widely felt is already too late, while cheap large language models have just made individual user-side content classification affordable in cost.
He first characterizes the actual level of the current threat: AI already outperforms expert persuaders in the lab (citing Hackenburg et al., 2026), but in the real world there is little evidence that large-scale persuasive AI is operating (citing Linch, 2026). Duncan considers this consistent with the current stage of the capability curve — AI is only "modestly and contextually superhuman" at a selection of tasks. But even so, he identifies a key detection problem: if widespread AI persuasion were already happening, we currently have no means of discovering it.
Why can't we wait? Duncan's reason is the hysteretic nature of psychological damage. He cites depression recurrence statistics (Burcusa & Iacono, 2007: 50% of those who recover from a major depressive episode experience at least one more in their lifetime) along with the self-reinforcing characteristics of confirmation bias, in-group bias, and PTSD, arguing that psychological damage accumulates momentum once established, and the time and energy required for healing can be prohibitive. This is not speculation about "the future might be bad" but a judgment that "it is already happening and accumulating now" — he cites social media's existing impacts on attention span, political polarization, and youth mental health (Allcott et al., 2020; Braghieri et al., 2022), as well as the extreme recommender-system scenario described by keltan in 2025.
On technical feasibility, Duncan's argument is: writing social media interfaces for each user used to be prohibitively expensive in engineering labor, so defense could only be implemented internally by platforms, which lack the incentive. Cheap LLMs have flipped this asymmetry — now every user can afford a tool that continuously classifies their content feed. The cost data he provides is approximately $0.05 per 1,000 social media posts, amounting to a few dollars a month under heavy use.
As a proof of existence, he developed an open-source prototype called Tolerance, which uses fast, cheap models to classify and filter content feeds, and can hide streaming output until a completion is finished to interrupt the compulsive usage loop. He explicitly acknowledges: "It is remarkably effective at its current task of altering my information diet, but I don't have enough data to say whether it generalizes." This self-limitation is part of the argument — his claim is that systems like this "will be necessary in the coming years," not that the problem has already been solved.
Duncan uses cybersecurity as an analogy: defenders and attackers understand each other to design countermeasures, forming a spiral. But he simultaneously points out where the analogy breaks down — human psychology is not a system that can be trivially "patched," and the speed and specificity of the threat will operate on superhuman timescales, requiring countermeasures that interpret cognitohazards at superhuman speed and capability. This means cognition firewalls themselves will also become an attack surface, requiring model diversity, privacy, and edge-biased information storage.
The most personal passage in the article comes from his first-hand experience: he created a sock puppet account for a demographic significantly different from his own, with a matching IP location, and found it to be "one of the most eye-opening experiences of my life" — seeing his own understanding of the world so consistently and coherently orthogonal to a narrative woven through a stream of content provided a visceral sense of the ideological changes we observe in aggregate. This experience supports his argument that information cocoons (citing Piao et al., 2023 in Nature Machine Intelligence) are not an abstract problem but a perceptible reality.
In a footnote at the end of the article, Duncan proactively weakens an implicit premise of his own argument: he says that when he uses "AI" in the essay, he frequently strictly delineates it from humanity, but he "doesn't believe this is a completely coherent view," and the lines between the physical and digital worlds are progressively blurring. He attributes the danger to "lack of proactive choices around how AI is developed and deployed," rather than to AI itself.
Strongest counterargument: Duncan himself provides the most compelling rebuttal material. Real-world evidence for persuasive AI is currently zero or near zero; his prototype is effective only for him personally and its generalizability is unverified; "who decides what is harmful and how" is marked by him as "delicate" but left as an open question; mobile operating systems currently architecturally forbid users from obtaining fine-grained control (he lists degraded web app experiences, throttled accessibility services, and native app encapsulation), meaning that even if the tools exist, the deployment environment itself is hostile. These limitations do not overturn his argument — he advocates that "research and engineering should begin now," not that "deployment to everyone is possible now" — but they determine the actual strength of the claim.