A Scriptable, Statistical Oracle for a Metadata Extraction System

An oracle is described for dynamic validation of an ap- plication (metadata extraction from scanned documents) where a moderate failure rate is acceptable provided that instances of failures during operation can be identified. The oracle combines a variety of deterministic tests and statis- tical tests based upon characteristics of the document col- lection on which the system operates. Because this system must adapt to a variety of document collections with differ- ent characteristics, a scripting language is developed that binds combinations of tests to the metadata fields expected in a given document collection. The suitability of the ora- cle is demonstrated by an experiment measuring its ability to mimic human judgments as to which of several alternate outputs for the same document would be preferred.