Simultaneous Object Detection, Tracking, and Event Recognition

Integrating information across modalities is a long-standing challenge for cognitive systems. The common internal structure and algorithmic organization of object detection, detection-based tracking, and event recognition facilitates a general approach to integrating these three components. This supports multidirectional information flow between these components allowing object detection to influence tracking and event recognition; and event recognition to influence tracking and object detection. The performance of the combination can exceed the performance of the components in isolation when inspecting the quality of the object tracks produced. We demonstrate this qualitatively on a number of videos which show how failures in each of the components are resolved when they are integrated together. This can be done with linear asymptotic complexity.