World Models: The Core of Physical AI Lies in Predicting the Next Frame, Not the Next Word
Byline Network reporter Oh Min-seon explains why world models—AI technologies that learn the laws of motion in reality from video data and predict the consequences of actions—are considered the core of physical AI. Using Genie 4 and NVIDIA Cosmos Policy as case studies, the article illustrates two applications: synthetic data generation and robotic action planning, while noting that the Sim-to-Real Gap and error accumulation remain fundamental unresolved obstacles.
In a demonstration released by Google DeepMind on October 27, 2026, for its Genie 4 system, a video was generated via text prompt: a crowned crane walks across a snowy landscape at dusk toward a white piano, pressing keys with its beak, accompanied by both piano notes and the sound of snow crunching. The technology behind this video is called a World Model. Its quality determines whether the scene is believable: if performance is insufficient, the sound may play before the beak touches the key, or the beak might pass directly through it; if performance is sufficient, the movement appears natural and physically consistent, resembling real footage.
A World Model can be defined as an AI that learns how the real world moves from data such as videos, thereby predicting what will happen next given a current state and an action. Byline Network reporter Oh Min-seon provides a clear analogy in her article: if language models like ChatGPT predict the next word, a World Model predicts the next frame. This distinction is critical for physical AI: robots need to understand not language, but the consequences of actions.
Physical AI—which enables entities like robots to act autonomously in the real world—relies on World Models because robots must understand basic physical laws such as gravity, velocity, and force. Oh Min-seon points out that the robotics industry currently applies World Models in two directions. First is generating training data: robots can learn not only from real-world collected data but also from synthetic data generated in virtual environments. Previously, these virtual environments were primarily built by developers manually inputting physical rules; the difference with World Models is that they learn these patterns directly from data rather than relying on human-defined rules, thus generating more realistic synthetic data. However, because they render scenes based on learned patterns rather than calculating physics, the results may look plausible but lack physical consistency. Second is planning actions during actual execution: Vision-Language-Action (VLA) models allow robots to decide what to do based on camera feeds and human instructions. By integrating a World Model, robots can virtually test the consequences of multiple actions before acting in reality, selecting the one most likely to succeed.