A Multimodal Dual-Stream Framework for Sheep Behavior Recognition Using Skeletal and Local Visual Fusion

Abstract

Simple Summary Monitoring the daily behaviors of sheep, such as standing, walking, and eating, is essential for managing their health and welfare in modern smart farms. However, automatically recognizing these behaviors in natural pastures is highly challenging because sheep often crowd together, blocking the camera’s view. Traditional artificial intelligence methods that only track the animals’ body joints often struggle in these situations, misinterpreting slight bodily twitches as actual walking due to a lack of environmental context. To solve this, we developed a novel artificial intelligence system that simultaneously analyzes both the movement of the sheep’s skeleton and the surrounding visual background. By combining these two streams of information, our system can accurately distinguish between similar behaviors even when the sheep are heavily obscured. Tested on real-world farm video data, our method achieved a high overall accuracy of over 93%, significantly reducing false alarms. This technology offers a reliable, non-contact solution for continuous sheep monitoring, supporting better animal welfare and more efficient farm management.