Pretrained Transformers for Effective and Efficient Information Access: BERT and Beyond

点击翻译按钮获取中文摘要

Users expect search systems that are fast (i.e., efficient) and return good results (i.e., effective), but these features are often in tension: effective in-depth content analysis can be slow, and fast systems often sacrifice quality. Building on a quarter of a century of experience developing techniques and building systems that connect users to relevant information, this proposal will lead to groundbreaking techniques for search and question answering (QA) that are effective as well as efficient. These capabilities will be demonstrated in high-impact applications on scientific texts such as the literature on coronaviruses or hydrology. Overall, my research will advance the scientific frontiers of both natural language processing (NLP) and information retrieval (IR). This proposal formulates search and question answering as ranking problems, and adopts an approach based on deep learning using a class of neural network architectures known as transformers. I will pursue the overall research vision from two complementary perspectives: (A) From the perspective of effectiveness, my research group will develop retrieval-specific self-supervision techniques and gain a better understanding of why transformers work as the basis for building improved ranking models. (B) From the perspective of efficiency, this research will build learned representations for ranking that are amenable to simple vector comparisons and develop model distillation techniques for accelerated inference. These separate threads will come together (C) in a principled framework for reasoning about effectiveness/efficiency tradeoffs, (D) demonstrated in applications that provide search and QA capabilities to literature in three scientific domains: biomedicine, hydrology, and artificial intelligence. Deployed prototypes will serve as a testbed for research and provide useful tools for stakeholders. Efforts along these lines have already begun: shortly after the start of the global COVID-19 pandemic, I led the development of Covidex (covidex.ai), an online and publicly accessible search engine for a collection of scientific articles related to coronaviruses. Such a system could be valuable, for example, to public health officials assessing the efficacy of different interventions and clinicians conducting meta-analyses. This work will achieve impact in four ways: (1) scientific innovations in the form of breakthroughs in deep learning methods that tackle search and QA, (2) computational artifacts in the form of open-source code, data, and models that will help foster adoption of the innovations arising from this research, (3) real-world applications for searching scientific literature in three domains, and (4) high-quality training opportunities. These efforts will contribute to the Canadian economy by enriching the ecosystem of ideas and talent around artificial intelligence and data science, complementing investments in these fields at the federal and provincial levels.