LeituraPesquisaRadar
Framework de investimento
Entrar / Cadastrar
← Voltar para a leitura
NotíciasTópico · 英伟达Cosmos 3Material5 min de leituraEditorial DisconfirmAI

NVIDIA Robotics Head Amit Goel: Physical AI Needs Three Computers; Dual-Tower Shared Latent Space Solves the Reasoning-Action Frequency Mismatch

Amit Goel, Head of Robotics Ecosystem and Edge AI Product at NVIDIA, explains on the Software Engineering Daily podcast: the fundamental architectural difference between physical AI and LLMs is that reasoning runs at low frequency while motor control runs at millisecond-level high frequency; the two must share context but cannot afford the bandwidth loss of decode-encode. NVIDIA's Cosmos 3 adopts a dual-tower design that exchanges embeddings directly in the latent space; meanwhile, training a world model requires three different computers — for model training, simulation-based reinforcement learning, and edge deployment.

2026-10-20 17:00 GMT+8

Amit Goel has been at NVIDIA for 15 years and currently heads the robotics ecosystem and the Edge AI product line, which is the Jetson family of computers. On episode 1958 of the Software Engineering Daily podcast, he explained to host Matt Merrill a fact that may be unfamiliar to LLM developers: training a physical AI model requires three completely different computers.

The first computer handles model training. Goel said the volume of data fed into a world model is orders of magnitude greater than what goes into a text LLM — just 15 minutes of video is several gigabytes. More critically, during training you need to retrieve images, motor control commands, and text annotations in parallel, creating storage and bandwidth demands that LLM pipelines have never faced. NVIDIA built a dedicated stack called a curator to handle this, because GPU resources are extremely scarce and precious and cannot be left idle waiting for CPU-based preprocessing to finish.

The second computer handles simulation. Goel explained that for a coding model, reinforcement learning verification means running the code, compiling it, and checking whether it errors out; for a world model, verification means calling a physics simulator, and that simulator requires ray-tracing cores for sensor simulation, demanding a different kind of GPU. Goel quoted a joke by Jim Fan, who leads NVIDIA's humanoid robotics research team, to capture the gap: Jim Fan says, 'I'm always envious of my friends in the LLM world — their problem is just not having enough GPUs. My problem is that half the sensor data is missing, the synchronization is off, after training I still have to find a simulation environment, and when I deploy to a real robot the cables are breaking.'

The third computer handles deployment. Goel emphasized that you cannot deploy on the same infrastructure you trained on. The model must undergo quantization, distillation, and compression to run on a small computer inside the robot. He cited Cosmos 3 Edge as an example: a 4-billion-parameter version of the Cosmos 3 family, designed specifically for the robot body, capable of running in real time on a computer called Jetson Thor. Goel described Jetson Thor as 'passport-sized, 2,000 teraflops of compute,' and noted that the 1X robot and the Boston Dynamics humanoid both use it.

Behind the split into three computers lies a core constraint Goel repeatedly stressed: the physical world's output is continuous, not discrete. He said a robot may simultaneously control 30 to 40 motors, each requiring millisecond-level continuous current output, rather than the token-by-token discrete prediction of text generation. This directly led to NVIDIA's 'dual-tower' architectural choice.

Goel described the dual-tower design as follows: a reasoning tower handles planning — based on current observations, past frames, and the objective function, it predicts the next sequence of actions, running at a lower frequency; an action tower generates the specific motor control commands, running at extremely high frequency. The two towers share the same latent space and exchange embeddings directly through cross-attention, rather than decoding one tower's output, sending it to the other, and re-encoding it. Goel said that if they were deployed separately, decode-encode would slow down bandwidth, and robotic systems need to operate in a closed loop, so NVIDIA chose to let both towers coexist within the same model, sharing context at high bandwidth.

Goel used an analogy to explain the division of labor: an architect plans what you need to build, and an executor carries out the construction according to the plan. Both need access to the same information, but what they do with that information is slightly different.

On the generalization problem, Goel said two paths are currently being pursued in parallel. The first is imitation learning: collecting real manipulation data in as many diverse environments as possible so the model's training distribution covers reality. But he acknowledged, 'However much data you collect in the real world, the real world has an infinite distribution.' The second path pairs a reasoning model with in-distribution data: the reasoning model learns common sense from text and video data on the internet, and when it encounters an out-of-distribution situation, it can determine 'this is closest to which known distribution,' then hand off to the action model for execution.

Goel also mentioned a unique capability of world models: counterfactual reasoning. He cited the Alpamayo work recently published by NVIDIA's self-driving car team, saying this reasoning model can predict 10 different futures, score each one, and select the safest option closest to the objective. He emphasized that this happens at evaluation time, without needing to wait until actual execution.

Finally, Goel discussed safety. He said the consequences of hallucination in physical AI are far more severe than in text, requiring multiple layers of protection: a conservative strategy (refuse to act when out of distribution), exposure to adversarial environments in simulation with scoring, guardrails at the planning stage (similar to LLM content safety filtering), and an external monitoring AI that observes and verifies the robot during execution.

All of this comes from Goel's own statements and represents NVIDIA's internal architecture design narrative. Goel explicitly said in the episode that 'there are many ways these models are being architected'; the dual-tower shared latent space is NVIDIA's choice and does not represent a converged solution across the field.

For software developers wanting to enter this space, Goel recommended starting with simulation: use an existing computer (a laptop with a 4090 GPU suffices), download a simulator and an off-the-shelf model, connect them, and see how it works. He mentioned that the partnership with Hugging Face has lowered the entry barrier — a few hundred dollars buys an SO-101 robotic arm, and with a $1,000 budget and a few days, you can get your first physical AI model running.

Fontes

  • softwareengineeringdaily.com↗
Tópico

英伟达Cosmos 3

英伟达于2026年5月31日正式发布并开源了NVIDIA Cosmos 3,这是其首个完全开放的“全模态世界模型”(Omnimodal World Model),旨在解决物理AI中感知、预测与行动分离导致的效率低下问题。【背景】传统机器人开…

6 etapas2 peçasAtualizado 2026-10-26 02:42

Como chegamos aqui

  1. 2026-05-31英伟达开发者博客正式宣布Cosmos 3发布,介绍其双塔MoT架构及Nano/Super模型规格,并开源代码与数据集。developer.nvidia.com ↗
  2. 2026-06-01英伟达在GTC台北大会上发布新闻稿,宣布成立“Cosmos联盟”,成员包括Agile Robots、Black Forest Labs等,强调其为全球首个完全开放的全模态物理AI基础模型。investor.nvidia.com ↗
  3. 2026-06-01GitHub仓库发布Cosmos 3正式版标签,确认Nano (16B) 和 Super (64B) 模型权重可用,并列出Text2Image、Image2Video等特定任务变体。github.com ↗

O que ainda está indefinido

  • Cosmos 3 Edge的具体发布日期及硬件兼容性列表尚未公布。
  • 在极端复杂动态环境(如拥挤街道或柔性物体操作)中,单次前向传播的动作生成准确率是否优于多步规划的传统管线?
  • 开源社区对Cosmos 3微调的实际采用率及衍生模型的质量如何?

Também nesta thread

  • Ming-Yu Liu, Head of Cosmos Research at NVIDIA: World Models Are a Toolkit, Not a Single Definition2026-10-25
Ler o tópico completo →
Continuar pesquisando →
Entrar / Cadastrar
LeituraPesquisaRadarFramework de investimento