Liao Hung-yuan, Director of the Institute of Information Science, Academia Sinica: Large Models Can Describe Images but Fail to Understand Human Actions; Modular Visual Understanding Is Needed to Bridge the Gap
Liao Hung-yuan, Director of the Institute of Information Science at Taiwan's Academia Sinica, explains that while current large vision-language models can generate image descriptions, they cannot determine if a person has fallen or understand relationships between objects. Academia Sinica’s approach is not to rebuild a large general-purpose model, but to develop modular visual capabilities—Behavior Analysis (Taiwan GBA) and Environmental Understanding (G-YOLO)—that work in tandem with large language models on edge devices, enabling AI to move from describing the world to understanding it.
When a photo of a child is input into a large vision-language model, the model’s description is merely "Everything is normal." Liao Hung-yuan, Director of the Institute of Information Science at Taiwan's Academia Sinica, uses this example to illustrate that while current general-purpose vision models possess considerable image-description capabilities, they struggle with more nuanced judgments closer to real-world needs—such as what the person is doing, whether danger is occurring, or what relationships exist between objects in the environment.
Liao Hung-yuan believes that to bridge this gap, Academia Sinica’s direction is not to build another large general-purpose model, but to develop specialized visual-understanding capabilities that complement large language models, including behavior analysis, object understanding, and edge computing. His reasoning is straightforward: edge devices have limited computing power and memory, making it impossible to run large models directly on cameras. Therefore, multimodal AI must adopt a modular approach, allowing different models to handle tasks suited to their strengths.
Home care is the most intuitive validation scenario for this architecture. When a camera detects someone falling, the system cannot simply know "there is a person in the frame"; it must determine if the person has fallen and requires assistance. If help is deemed necessary, the system uses text-to-speech to ask, "Do you need help?" After the user responds, automatic speech recognition interprets the voice content, which is then connected to a large language model. Finally, the system notifies family members or emergency services while simultaneously relaying the result back to the person who fell.
The core of behavior analysis is Taiwan GBA, developed by Academia Sinica in collaboration with Chung Yuan Christian University and National Tainan University. Its key concept is the Universal Action Space (UAS), inspired by DeepMind’s Kinetics-600 dataset—which collected videos of 600 types of human actions, each lasting about 5 to 10 seconds. The research team used these videos to establish an action feature space, enabling the model to learn correlations between different actions. In the future, new behaviors such as falling, waving, or attacking do not require building completely independent models; instead, they can be mapped into the existing action space. Liao Hung-yuan describes UAS as a "comprehensive sports stadium": different sports or behaviors can all be understood within this space.
However, human behavior cannot be judged from a single image; it requires understanding temporal changes. Two photos showing a person standing versus lying on the ground may appear as different states, but what truly matters is the process "from standing to falling." The research team utilizes Video Swin Transformer to analyze dynamic features in videos, identifying locations in the images related to actions—for instance, when playing tennis, the model focuses on the person’s hands, racket, and body movement areas—and converts these dynamic details into trainable behavior features, mapping them into the universal action space.
In home care applications, the research team tested using Nanyang Technological University’s NTU RGB+D dataset, which contains 220 types of human actions. The team selected nine care-related actions for analysis. Liao Hung-yuan stated that in the dataset testing environment, the model’s recognition rate reaches approximately 80% to 90%. However, he also acknowledged that the biggest challenge in actual deployment is not laboratory data, but real-world data. Many crime datasets consist of simulated scenarios; for example, someone holding a knife might be acting, differing significantly from a real crime scene. For AI to be truly applied on-site, more real-world data accumulation is still required.
Regarding environmental understanding, Academia Sinica researcher Dr. Wang Chien-yao extended the YOLO v7 and YOLO v9 architectures to develop G-YOLO (Generalist YOLO). Traditional YOLO excels at real-time object detection, quickly answering "what things are in the picture," such as identifying people or cars in a street view. But for self-driving cars or robots, what is truly needed is understanding relationships between objects: if a group of people is crossing the road, AI must not only know "there are people" but also understand that this group is crossing the road, identify nearby buildings, and comprehend the relationships between people and surrounding objects. With G-YOLO added, the model can generate more complete scene descriptions, such as noting that approximately 15 people are crossing the road, with shops, buildings, and road facilities nearby.
Currently, Taiwan GBA can recognize 12 behaviors, including falling, fighting, and brandishing knives, covering application scenarios in home care, public safety, and campus security. G-YOLO continues to expand depth estimation and multi-view understanding capabilities, enabling AI to know not just "what was seen," but also "where objects are, how far away they are, and how they interact with each other."
Liao Hung-yuan believes that in the future, self-driving cars, robots, and smart care systems will require not only large language models but also a set of multimodal capabilities that can understand the real world. Large models allow AI to begin describing the world, but to truly enter the world, AI still needs to understand human actions, environmental changes, and the meaning behind events. This is the core direction of Academia Sinica’s investment in autonomous multimodal AI research and development.