Ming-Yu Liu, Head of Cosmos Research at NVIDIA: World Models Are a Toolkit, Not a Single Definition
Ming-Yu Liu, head of Cosmos research at NVIDIA, explained on the Machine Learning Street Talk podcast how Cosmos 3 combines vision-language models with diffusion generators and proposed that 'world models' are a set of co-trained tools rather than a single definition.
Cosmos 3 can describe videos, generate videos, and output robot actions. Ming-Yu Liu introduced its architecture to host Tim: a vision-language model reasons one token at a time, and its weights then initialize a bidirectional diffusion generator for video, audio, and action generation; a shared temporal position encoding aligns signals running at different rates.
Ming-Yu Liu views 'world models' as a set of tools rather than a single definition: forward dynamics, inverse dynamics, and policy are co-trained under capacity constraints, helping each other. He noted that the most practical clue is testing—neural simulators do not need accurate success rates, only the ability to rank Policy A above Policy B as the real world would, allowing teams to filter checkpoints worthy of real-world validation. Cosmos Dreams applies this closed-loop approach to driving and robotics scenarios.
This episode is paid partnership content from NVIDIA. All viewpoints above come from the program's summary of Liu's discussion, not a verbatim transcript or independent reproduction results. Liu's specific reasoning processes, limiting conditions, and unmentioned failure cases are not visible in the summary. Full technical details of Cosmos 3 are contained in its paper (arXiv:2606.02800), which this summary does not cover.