Retrieving what's relevant in audio and video: statistics and linguistics in combination

We present some of the technology developed at StreamSage for indexing and retrieving audio/video data. A primary difficulty of this task is precise extraction of the passages relevant to the query from the audio/video stream, which is crucial in presenting results in a manageable fashion, particularly locating their beginning and end. We focus on the combination of linguistic and statistical approaches employed to construct content-specific relevance intervals in timed media. These techniques, including topic and topic boundary identification, referent resolution, and large coverage word sense disambiguation, must be automatic, scalable, and domain-independent.