Preprint Proposes 'Object Retargeting' to Turn Phone-Captured Human Demos into Cross-Embodiment Robot Post-Training Data
New research suggests recording object trajectories instead of hand motions allows smartphone-captured human demos to serve as post-training data for diverse robots, achieving a 31.2% simulation pass rate.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A new method proposed by Nagasrikanth Kallakuri demonstrates that human manipulation videos recorded with smartphones can serve as post-training data for robots with different physical structures. The core innovation lies in retargeting the motion of the manipulated object rather than the human hand.
Traditionally, human demonstrations are used primarily for pretraining vision-language-action models, with limited applicability to target-embodiment or task-oriented training. The new system extracts the object's pose over time, contact intervals, and a reconstructed 3D model to create a 'body-free' record. Any robot, provided with a URDF model and a short descriptor, can be solved against this record and evaluated in simulation using the demonstrated object motion as ground truth.
In an analysis of 42 captures across 8 robots comprising 157 pairs, the method achieved a 31.2% pass rate in simulation, with 52.2% failures and 14.0% refusals based on kinematic feasibility checks. Notably, every pair flagged as infeasible but forced to run failed. No policy has yet been trained on these records, leaving real-world execution efficacy to downstream validation.