Understanding Images of Graphical User Interfaces: A New Approach to Activity Recognition for Visual Surveillance

A fundamental problem in surveillance systems is the specification of "activities of interest". While various activity recognition systems have been developed, they have used complex hand-coded representations. What is of interest to a particular surveillance system user can vary greatly, and the security forces using the system are not, in general, advanced computer users. This paper presents a novel paradigm for specifying and recognizing activities in a surveillance system by visualizing tracking results, and using computer vision techniques to interpret the images presented by their Graphical User Interfaces. Representation of activities of interest can be easily drawn by users. Not only is the drawing-based specification of activities of interest easier than previous approaches, but when a "rule" fires, it is easy to explain "why", by showing the operator the associated drawings. This approach also permits a new type of system integration, where we integrate the "display" of sensors rather than trying to develop a complex communication protocol. A user study is reviewed showing the proposed approach is thousands of time faster than HMMs for event definition while it is simultaneously more accurate in event recognition.