Progress in the Application of Natural Language Processing to Information Retrieval Tasks

Techniques of automatic natural language processing have been under development since the earliest computing machines, and in recent years these techniques have proven to be robust, reliable and efficient enough to lead to commercial products in many areas. The applications include machine translation, natural language interfaces and the stylistic analysis of texts but NLP techniques have also been applied to other computing tasks besides these. In this paper we will examine and review recent progress in using the lexical, syntactic, semantic and discourse levels of the language analysis for tasks like automatic and semi-automatic indexing of text, text retrieval, text abstracting and summarisation, thesaurus generation from text corpus and conceptual information retrieval. Our own work on the application of syntactic analysis to the matching and ranking of phrases using structured representations of texts, will be included in the overview. Finally, the prospects for gains in terms of overall retrieval effectiveness or quality will be discussed.