MIT and NYU Researchers Develop FLEX System Using GPT-4o to Infer General Robot Preferences from Short Commands

Researchers at the Massachusetts Institute of Technology (MIT) and New York University have developed an AI-assisted robot learning system called FLEX (Flexible Feature Editing Pipeline). The system addresses a key limitation in conventional robot motion planning: while a robot may understand a specific command, it often fails to recognize the underlying reason or generalize the constraint to similar situations. For example, if told not to spill coffee on a laptop, a standard robot might not avoid spilling on other nearby electronic devices unless explicitly instructed.

FLEX modifies the internal representation a robot uses when planning its movements by employing three operations: Add, Generalize, and Mask. It uses OpenAI’s GPT-4o language model to identify relationships between objects and interpret the intentions behind human corrections. If a robot carrying a bowling ball is warned not to drop it on a mug, the system infers that fragile objects should be avoided generally, rather than treating the mug as the only potential hazard. Similarly, a warning about carrying a lit candle near paper can be generalized into a preference for avoiding flammable materials.

The research was led by Andreea Bobu of MIT’s Computer Science and Artificial Intelligence Laboratory, with Audrey Lee, Nathaniel Dennler and Ilia Sucholutsky. The team evaluated the approach in two-dimensional navigation environments and simulated tabletop manipulation tasks involving a seven-degree-of-freedom Franka robotic arm. In sequential experiments, FLEX maintained higher normalized reward scores than approaches restricted to adding or removing features, which could accumulate unnecessary constraints or fail to generalize earlier instructions.