NVIDIA Demonstrates a Phased Workflow for AI Agent Collaborative Training of Robots
The NVIDIA team used three AI agents—Hermes, Nemotron 3.5 Bolt, and Astra—to start from a single prompt and sequentially complete simulation environment planning, 3D asset creation, Isaac Sim environment assembly, Isaac Lab reinforcement learning training (with 4,096 parallel environments), and local policy validation, demonstrating the specific practices for structured handoff packages between multi-agents.
Teaching a simulated robot to navigate an obstacle course requires multiple stages: environment planning, 3D asset modeling, physics simulation assembly, policy training, and validation. The NVIDIA team demonstrated how three AI agents divide labor to complete this pipeline.
The process begins with a natural language prompt. On an RTX Spark local development machine, Hermes and Nemotron 3.5 Bolt translate the prompt into a step-by-step project plan, including task definition, resource inventory, expected outputs, and instructions for each stage. Once the plan is complete, Hermes packages all outputs into a handoff package: the project plan, a summary and source links for robot documentation research, concept images, an asset inventory (including identifiers, visible feature descriptions, and crop reference images), and execution instructions for the next stage. Hermes then automatically delegates the work to Astra.
Upon receiving the handoff package, Astra begins creating 3D assets. It first uses cloud image editing to enhance low-resolution objects cropped from the concept image into clear references, then compares the enhanced images with the original crops to confirm consistency in silhouette, proportion, and color. Dimensions and hidden surfaces that cannot be determined from the images are recorded as modeling assumptions. Astra then directs FreeCAD to construct the base geometry, uses Blender to add details and materials, and finally exports USD assets.
The training phase is conducted in the cloud using Isaac Lab. This experiment starts with the Microduck example environment from Isaac Lab, performing additional training on a pre-trained walking policy for traversing ramps and obstacles. Using head-camera depth and robot state as observation inputs, it runs on an NVIDIA RTX PRO 6000 Blackwell Workstation GPU with 4,096 parallel environments. The trained policy returns to RTX Spark for local validation in Isaac Sim, checking obstacle detection, movement adjustment, and track completion.