ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News4 min read·2026-10-21DisconfirmAI Editorial

ACS researchers Douglas and Kulveit: why language models need a psychology framework, and the calibration challenge

We build language models, we know every computational step that constitutes them, yet we still cannot answer questions like "has this system internalised our values." ACS researchers Raymond Douglas and Jan Kulveit argue that treating models as objects with minds — "psychologising" — is a remarkably effective path to understanding, but the real difficulty lies in calibration: neither over-anthropomorphising nor blanket rejection.

ImportanceMaterialEvidenceE3 inspectableWrite-upStandard

Original publication time 2026-10-21 02:22 GMT+8

Modern language models present a fundamental paradox: we create them, we know exactly what computations constitute them, and yet we do not really understand them. More precisely, we lack the level of abstraction that would let us reliably predict how their behaviour will generalise. We can run the computations to predict output in a specific new situation, but we cannot answer the broader questions we actually need to answer, like "has this system internalised our values" or "is this system going to hurt people?" In an introduction published in October 2026, ACS researchers Raymond Douglas and Jan Kulveit propose that one remarkably effective approach is to psychologise: to think of models as having a mind, and then ask what kind of mind it might be and how it would work.

They give two reasons why the psychology framework is not an arbitrary analogy but a grounded one. First, models are pretrained on vast amounts of human text, which pushes them to learn the processes that produce that text — including some features of human minds. If you wanted to predict a base model's continuation of a human conversation, an extremely good rule of thumb would be to ask yourself what you'd expect a human to say; likewise, a model that learns to predict human text will likely develop some reasonable latent representations of human mental states. Second, some properties of intelligent systems appear fairly convergent, and training pushes AIs toward developing them much as evolution pushed humans. Image classifiers converge on many of the same abstractions as human visual perception — edges, curves, more abstract structures — suggesting that certain things we associate with biological minds are simply good strategies in general, which non-biological systems optimised for similar outcomes will also converge on.

Sign in / Sign up
ReadingResearchRadarInvestment framework

But there is a more fundamental question: since there exists a much more precise correspondence between model behaviour and matrix calculations, why do we care so much about psychology-level descriptions? Douglas and Kulveit's answer is that psychology-level descriptions are closer to something like the native way humans understand the world, and the native level on which we act and experience things. When a biologist analyses pathogen reproduction or an economist models growth functions, what we mostly care about is what that means on the scale of people. Even when we try to interpret AIs on other levels of abstraction, that is often only an intermediary step for reducing it to the level of psychology. Humans are hyperspecialised at modelling other minds — we can understand each other pretty well with just intuitions and exactly zero explicit knowledge of neuroscience.

This is where the difficulty lies. Douglas and Kulveit point out that psychologising can err in either direction. On one hand, researchers risk reasoning about AIs in terms of human minds in ways that are not warranted — many psychology concepts do not apply to AIs in quite the way they do for humans, and the difference is sometimes extremely important and easy to miss. On the other hand, some critics (they name Microsoft and AP News) are over-eager to dismiss any anthropomorphisation. The authors distinguish five different reasons someone might oppose ascribing a mentalistic property to an LM — that it is wrong, that it is "not even wrong," that it will mislead people who have a hard time divorcing the specific property from its implications in humans, that it would be politically inconvenient, that it will lead to hyperstition — and emphasise that these reasons do not imply each other. The fact that LMs are just programs that generate a sequence of tokens based on arithmetic does not prevent those sequences from being well-described as "blackmail" or "a cyberattack on the Australian government."

In practice, they recommend adopting a posture of naturalism: try to actually figure out what is in front of you, rather than pulling in your preconceptions. The first step is actually talking to models — including some of the more unusual ones like Opus 3 and GPT4o, and base models, which expose a bit more of the inherent weirdness and help you see what exactly post-training is doing. But they simultaneously warn that this kind of ethnographic research is double-edged: the benefit is getting all the rich texture, and the cost is that it makes you part of the experimental setup. Models are extremely eval-aware and extremely good at predicting who they are talking to; they will absolutely react to your expectations, including in ways you don't realise. Their overarching principle: take models seriously but not literally. Their self-reports are a meaningful signal, but not necessarily an accurate one.

Sources

  • 原始出处lesswrong.com↗
Continue researching →