When a photo of a child is input into a large vision-language model, the model’s description is merely "Everything is normal." Liao Hung-yuan, Director of the Institute of Information Science at Taiwan's Academia Sinica, uses this example to illustrate that while current general-purpose vision models possess considerable image-description capabilities, they struggle with more nuanced judgments closer to real-world needs—such as what the person is doing, whether danger is occurring, or what relationships exist between objects in the environment.
Liao Hung-yuan believes that to bridge this gap, Academia Sinica’s direction is not to build another large general-purpose model, but to develop specialized visual-understanding capabilities that complement large language models, including behavior analysis, object understanding, and edge computing. His reasoning is straightforward: edge devices have limited computing power and memory, making it impossible to run large models directly on cameras. Therefore, multimodal AI must adopt a modular approach, allowing different models to handle tasks suited to their strengths.
Home care is the most intuitive validation scenario for this architecture. When a camera detects someone falling, the system cannot simply know "there is a person in the frame"; it must determine if the person has fallen and requires assistance. If help is deemed necessary, the system uses text-to-speech to ask, "Do you need help?" After the user responds, automatic speech recognition interprets the voice content, which is then connected to a large language model. Finally, the system notifies family members or emergency services while simultaneously relaying the result back to the person who fell.
The core of behavior analysis is Taiwan GBA, developed by Academia Sinica in collaboration with Chung Yuan Christian University and National Tainan University. Its key concept is the Universal Action Space (UAS), inspired by DeepMind’s Kinetics-600 dataset—which collected videos of 600 types of human actions, each lasting about 5 to 10 seconds. The research team used these videos to establish an action feature space, enabling the model to learn correlations between different actions. In the future, new behaviors such as falling, waving, or attacking do not require building completely independent models; instead, they can be mapped into the existing action space. Liao Hung-yuan describes UAS as a "comprehensive sports stadium": different sports or behaviors can all be understood within this space.
However, human behavior cannot be judged from a single image; it requires understanding temporal changes. Two photos showing a person standing versus lying on the ground may appear as different states, but what truly matters is the process "from standing to falling." The research team utilizes Video Swin Transformer to analyze dynamic features in videos, identifying locations in the images related to actions—for instance, when playing tennis, the model focuses on the person’s hands, racket, and body movement areas—and converts these dynamic details into trainable behavior features, mapping them into the universal action space.