Parsing ink annotations on heterogeneous documents

Annotation is an integral part of reading, comprehending, commenting, and authoring notes and documents. In this paper we present a system for recognizing annotations in a flexible digital notebook that may contain a variety of content ranging from text, to images, to handwritten notes. To accomplish the recognition task in real-time makes the complicated annotation parsing problem more difficult. Our approach differs from previous approaches in several ways. First, our approach handles annotations on ink notes, which are significantly more ambiguous than annotations on printed documents and hence more difficult to recognize. Second, our approach is entirely learned from data, so it is easy to adapt to other scenarios. Third, our approach is more thoroughly evaluated than previous systems. On a test set of real user notes, the system has achieved an average recall of 0.9258 on all annotation types. Finally, the implementation of the approach will be commercially available as an API in the upcoming release of Windows® Vista® and Office 12®.