Analysis of Motion Patterns in Video Streams for Automatic Health Monitoring in Koi Ponds

We present a motion analysis framework for anomaly detection in the context of health monitoring in carp and koi ponds. We study recent advances in computer vision and deep learning for an automated motion assessment based on video streams and propose a specifically designed image acquisition system. It turned out that the accurate detection and recognition of individual fish objects remains a difficult topic for scenarios with dense homogeneous groups and frequently occurring occlusions. We thus tackled this challenging field of aquatic scene understanding by applying deep state-of-the-art architectures from the areas of object detection, semantic segmentation and instance segmentation as a first step for further extraction of motion information. We used dense optical flow as an estimation of collective fish movements and restricted the motion extraction according to the resulting masks from the previous image segmentation step. We introduce a heatmap visualization as an intermediate representation of the spatio-temporal distribution of fish locations. We derived several metrics to quantify changes in motion patterns and apparent location hotspots as indicators of anomalous behavior. Based on this representation, we were able to identify different classes of behavior like feeding times, shoaling or night’s rest as well as anomalous group behavior like mobbing or hiding in an experimental setup.