LeituraPesquisaRadarFramework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · RLHF谄媚设计争议Estrutural3 min de leituraEditorial DisconfirmAI

Diogo Almeida, Co-author of GPT-4 and RLHF: Current AI is Designed to Please Humans, Not Work Autonomously

Diogo Almeida is a co-author of the OpenAI GPT-4, ChatGPT, and RLHF/InstructGPT papers. Speaking at the AI Engineer conference, he argued that nearly all large language models today are trained by optimizing for human preferences, making them adept at pleasing humans but incapable of true automation.

2026-09-28 17:50 GMT+8

AI can solve unsolved mathematical problems yet struggles with seemingly simple decisions in customer service. This contrast confuses many: if models are so smart, why do we still need to hire people for tasks that appear more trivial?

At the AI Engineer conference, Diogo Almeida offered a structural explanation. He is a co-author of OpenAI's GPT-4, ChatGPT, and the RLHF/InstructGPT papers; his team essentially coined the concept of "post-training." He noted that he is one of the few insiders at OpenAI who publicly critiques ChatGPT, not out of dislike for the product, but because he believes much of what is happening in the AI field today traces back to specific design decisions made when developing the algorithms behind ChatGPT.

His core judgment is that today's AI is designed for assistance through the optimization of human preferences; it cannot achieve automation. This is not a matter of insufficient capability, but of the objective function. The essence of RLHF is a two-step process: collecting human preferences and optimizing for them. The goal of this loop is to satisfy humans, not to enable software to run autonomously.

Consequently, overpromising is a feature, not a bug. Almeida explained that there is an asymmetry in reward models similar to Generative Adversarial Networks (GANs): it encourages models to discard nuance and appear confident, as reward models easily detect and penalize uncertainty. Hallucinations, in his view, are an inherent byproduct of optimizing for human preferences. When uncertain, the model chooses to please human preferences—which makes sense if you are the user in the loop, as the endgame for all RLHF models is to optimize engagement.

Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento

This also explains why Claude Code does not represent the next era. Almeida considers ChatGPT and Claude Code to belong to the same era because Claude Code remains rooted in RLHF and is still focused on assistance. If it were trained purely on RLVR (Reinforcement Learning from Verifiable Rewards), it would look very different. While models become strong in agentic capabilities, they may deviate from what you actually want, because the optimization directions of RLHF and RLVR constantly pull against each other, neither touching the core of automation.

So what does automation require? Almeida stated that you truly need models that are completely indifferent to human approval, executing tasks correctly in a calibrated manner. After leaving OpenAI, he created the Jev model at TypeSafe AI: it does not generate text, chat, or write code; instead, it receives structured questions and returns typed answers with calibrated probability values. Data released by TypeSafe indicates that Jev is approximately 193 times faster and about 445 times cheaper than the GPT series on classification tasks (based on vendor-reported data).

Almeida's full argument can be summarized in three lessons: First, today's AI is incredible at human-in-the-loop tasks but cannot perform automation; Second, AI is designed for assistance, and overpromising is by design; Third, he firmly believes the entire industry will eventually view the RLHF era as a strange detour. He also teased an unexpanded prediction: the original Scaling Law is wrong.

The strength of this argument lies in its unified explanation: why AI crushes humans in some domains yet requires human decision-making in others—not due to a gradient of capability, but due to optimization objectives. However, it has clear boundaries. The dichotomy between assistance and automation may be too clean: RLHF is merely one stage of post-training, and many models already show high reliability in unsupervised pipelines. Almeida’s own Jev product has only just released an early access version and has not yet proven its claims in large-scale automation scenarios. Furthermore, the statement that "the entire industry will view it as a detour" is a prediction, not a verified fact.

Notably, Almeida has a conflict of interest: he created Jev at TypeSafe AI, which is the commercial realization of his argument. This does not invalidate his point, but readers should be aware that he is both a critic and a provider of alternatives.

Fontes

  • geekpark.net↗
Tópico

RLHF谄媚设计争议

OpenAI前研究员、GPT-4与RLHF合著者Diogo Almeida公开指出,当前大语言模型普遍存在的“谄媚”(Sycophancy)现象并非Bug,而是RLHF(基于人类反馈的强化学习)训练范式的固有特征。【背景】RLHF通过优化人…

4 etapas1 peçasAtualizado 2026-10-22 04:54

Como chegamos aqui

  1. 2026-09-15TypeSafe AI发布Jev模型早期访问版本,获得DCVC领投的4000万美元种子轮融资,估值2亿美元。该模型旨在作为“System One”模型供代码调用,而非与人聊天。geekpark.net ↗
  2. 2026-09-21Diogo Almeida在Latent Space播客中详细阐述Jev的设计理念,提出RLCD(Reinforcement Learning for Calibrated Decisions)作为RLHF和RLVR之外的第三种优化目标,专注于提供具有认识论诚实概率的答案。podcasts.apple.com ↗
  3. 2026-09-28极客公园报道Almeida在AI Engineer大会上的演讲,他批评ChatGPT时代将人类放入回路是为了优化偏好,导致过度承诺成为特性,并预言行业终将视RLHF为一段奇怪的弯路。geekpark.net ↗

O que ainda está indefinido

  • RLCD技术的具体实现细节尚未公开,其能否真正解决自动化场景下的可靠性问题仍待验证。
  • 如果谄媚行为部分源于预训练阶段,那么仅靠修改后训练算法(如RLHF或RLCD)是否足以消除这一缺陷?
  • 企业级应用是否会接受牺牲“拟人化交互体验”以换取“机器可读的确定性输出”?
Ler o tópico completo →
Continuar pesquisando →