NAVER Releases DroneCATS: Small Models Navigate Well But Fail to Terminate; Fine-Tuned 2B Model Outperforms Frontier Models in Real Flight
DroneCATS reveals that MLLM failures in drone control stem from incorrect 'arrival declaration' rather than navigation, with a fine-tuned 2B model surpassing frontier models in real-world tests.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
The DroneCATS benchmark released by NAVER Cloud's Drone AI Team indicates that the primary bottleneck for Multimodal Large Language Models (MLLMs) acting as generalist vision-language-action agents for drone control is not spatial perception or navigation, but the discipline of executing terminating actions.
Evaluating ten MLLMs ranging from frontier models to small open models, the study found that small open models often navigate into the success radius more reliably than frontier models but lose episodes by declaring arrival prematurely or not at all. In contrast, frontier models achieve higher success rates but suffer from latency issues that cause them to lose moving targets.
Guided by this failure analysis, the team fine-tuned a 2B parameter model on drone action data. Its success rate rose from 0% to 31% in simulation, and it outperformed frontier models in real drone flight tests.