Deep information retrieval - diving into information granularity and abstraction

点击翻译按钮获取中文摘要

Although the final goal of information retrieval (IR) is to find relevant (pieces of) information, the research in IR has been generally limited to retrieving entire documents. In practice, many user queries are intended to find more precise segments of texts within documents. The first problem we deal with in this research is to make it possible for end users to find answers of different levels of granularity.***IR also uses a unique representation for a document. In practice, user's search intents vary greatly: a query may be used to find a text containing a specific sequence of words, or a text containing the required semantics. The unique representation is unable to satisfy such various search intents. We propose to represent texts at different levels of abstraction, from surface words to more abstract semantic representations. Queries with different intents could then be compared with text representations at appropriate levels of abstraction.***Finally, IR needs a complex ranking function, which is usually learned from ranking examples (learning-to-rank). However, the learning-to-rank approaches have never been used in combination with learning representations. We propose to combine the two learning tasks that are required. In addition, the matching function may also involve the use of domain knowledge to infer the semantic relations between a document and a query.***The above problems will be investigated using deep learning techniques. Neural network representations for texts will be created at several layers, corresponding to different levels of abstraction. In addition, both the entire document and the searchable segments in it are represented. The learning-to-rank method will be trained to select the appropriate way to use the representations created in the network to rank documents for a query.***The ultimate goal of this research is to develop more intelligent IR systems that can understand and cope with various user information needs.