NVIDIA Robotics Head Amit Goel: Physical AI Needs Three Computers; Dual-Tower Shared Latent Space Solves the Reasoning-Action Frequency Mismatch
Amit Goel, Head of Robotics Ecosystem and Edge AI Product at NVIDIA, explains on the Software Engineering Daily podcast: the fundamental architectural difference between physical AI and LLMs is that reasoning runs at low frequency while motor control runs at millisecond-level high frequency; the two must share context but cannot afford the bandwidth loss of decode-encode. NVIDIA's Cosmos 3 adopts a dual-tower design that exchanges embeddings directly in the latent space; meanwhile, training a world model requires three different computers — for model training, simulation-based reinforcement learning, and edge deployment.
Amit Goel has been at NVIDIA for 15 years and currently heads the robotics ecosystem and the Edge AI product line, which is the Jetson family of computers. On episode 1958 of the Software Engineering Daily podcast, he explained to host Matt Merrill a fact that may be unfamiliar to LLM developers: training a physical AI model requires three completely different computers.
The first computer handles model training. Goel said the volume of data fed into a world model is orders of magnitude greater than what goes into a text LLM — just 15 minutes of video is several gigabytes. More critically, during training you need to retrieve images, motor control commands, and text annotations in parallel, creating storage and bandwidth demands that LLM pipelines have never faced. NVIDIA built a dedicated stack called a curator to handle this, because GPU resources are extremely scarce and precious and cannot be left idle waiting for CPU-based preprocessing to finish.
The second computer handles simulation. Goel explained that for a coding model, reinforcement learning verification means running the code, compiling it, and checking whether it errors out; for a world model, verification means calling a physics simulator, and that simulator requires ray-tracing cores for sensor simulation, demanding a different kind of GPU. Goel quoted a joke by Jim Fan, who leads NVIDIA's humanoid robotics research team, to capture the gap: Jim Fan says, 'I'm always envious of my friends in the LLM world — their problem is just not having enough GPUs. My problem is that half the sensor data is missing, the synchronization is off, after training I still have to find a simulation environment, and when I deploy to a real robot the cables are breaking.'
The third computer handles deployment. Goel emphasized that you cannot deploy on the same infrastructure you trained on. The model must undergo quantization, distillation, and compression to run on a small computer inside the robot. He cited Cosmos 3 Edge as an example: a 4-billion-parameter version of the Cosmos 3 family, designed specifically for the robot body, capable of running in real time on a computer called Jetson Thor. Goel described Jetson Thor as 'passport-sized, 2,000 teraflops of compute,' and noted that the 1X robot and the Boston Dynamics humanoid both use it.
Behind the split into three computers lies a core constraint Goel repeatedly stressed: the physical world's output is continuous, not discrete. He said a robot may simultaneously control 30 to 40 motors, each requiring millisecond-level continuous current output, rather than the token-by-token discrete prediction of text generation. This directly led to NVIDIA's 'dual-tower' architectural choice.
Goel described the dual-tower design as follows: a reasoning tower handles planning — based on current observations, past frames, and the objective function, it predicts the next sequence of actions, running at a lower frequency; an action tower generates the specific motor control commands, running at extremely high frequency. The two towers share the same latent space and exchange embeddings directly through cross-attention, rather than decoding one tower's output, sending it to the other, and re-encoding it. Goel said that if they were deployed separately, decode-encode would slow down bandwidth, and robotic systems need to operate in a closed loop, so NVIDIA chose to let both towers coexist within the same model, sharing context at high bandwidth.
Goel used an analogy to explain the division of labor: an architect plans what you need to build, and an executor carries out the construction according to the plan. Both need access to the same information, but what they do with that information is slightly different.
On the generalization problem, Goel said two paths are currently being pursued in parallel. The first is imitation learning: collecting real manipulation data in as many diverse environments as possible so the model's training distribution covers reality. But he acknowledged, 'However much data you collect in the real world, the real world has an infinite distribution.' The second path pairs a reasoning model with in-distribution data: the reasoning model learns common sense from text and video data on the internet, and when it encounters an out-of-distribution situation, it can determine 'this is closest to which known distribution,' then hand off to the action model for execution.
Goel also mentioned a unique capability of world models: counterfactual reasoning. He cited the Alpamayo work recently published by NVIDIA's self-driving car team, saying this reasoning model can predict 10 different futures, score each one, and select the safest option closest to the objective. He emphasized that this happens at evaluation time, without needing to wait until actual execution.
Finally, Goel discussed safety. He said the consequences of hallucination in physical AI are far more severe than in text, requiring multiple layers of protection: a conservative strategy (refuse to act when out of distribution), exposure to adversarial environments in simulation with scoring, guardrails at the planning stage (similar to LLM content safety filtering), and an external monitoring AI that observes and verifies the robot during execution.
All of this comes from Goel's own statements and represents NVIDIA's internal architecture design narrative. Goel explicitly said in the episode that 'there are many ways these models are being architected'; the dual-tower shared latent space is NVIDIA's choice and does not represent a converged solution across the field.
For software developers wanting to enter this space, Goel recommended starting with simulation: use an existing computer (a laptop with a 4090 GPU suffices), download a simulator and an off-the-shelf model, connect them, and see how it works. He mentioned that the partnership with Hugging Face has lowered the entry barrier — a few hundred dollars buys an SO-101 robotic arm, and with a $1,000 budget and a few days, you can get your first physical AI model running.