Pretrained Transformers for Effective and Efficient Information Access: BERT and Beyond

用户期望快速的搜索系统（即，有效）并返回良好的结果（即，有效），但这些功能通常处于紧张状态：有效的深入内容分析可能会很慢，快速的系统通常会牺牲质量。基于世纪来开发技术和构建将用户与相关信息联系起来的系统的经验，该提案将导致有效且高效的搜索和问答（QA）的突破性技术。这些能力将在科学文本的高影响力应用中得到展示，例如关于冠状病毒或水文学的文献。总的来说，我的研究将推进自然语言处理（NLP）和信息检索（IR）的科学前沿。该提案将搜索和问题回答公式化为排名问题，并采用基于深度学习的方法，使用一类称为transformers的神经网络架构。我将从两个互补的角度追求整体研究愿景：（A）从有效性的角度来看，我的研究小组将开发检索特定的自我监督技术，并更好地理解为什么变压器作为构建改进的排名模型的基础。(B)从效率的角度来看，这项研究将建立学习表示的排名是服从简单的向量比较和开发模型蒸馏技术加速推理。这些独立的线索将在一个原则性的框架中结合在一起（C），用于推理有效性/效率权衡，（D）在为三个科学领域的文献提供搜索和QA功能的应用程序中得到证明：生物医学，水文学和人工智能。部署的原型将作为研究的试验平台，并为利益相关者提供有用的工具。沿着这些路线的努力已经开始：在全球COVID-19大流行开始后不久，我领导开发了Covidex（covidex.ai），这是一个在线和可公开访问的搜索引擎，用于收集与冠状病毒相关的科学文章。例如，这样一个系统可能对公共卫生官员评估不同干预措施的有效性和临床医生进行荟萃分析很有价值。这项工作将从四个方面产生影响：（1）以深度学习方法的突破为形式的科学创新，解决搜索和QA问题，（2）以开源代码、数据和模型为形式的计算工件，这将有助于促进采用这项研究产生的创新，（3）在三个领域搜索科学文献的现实应用，（4）高质量的培训机会。这些努力将通过丰富围绕人工智能和数据科学的思想和人才生态系统，补充联邦和省级对这些领域的投资，为加拿大经济做出贡献。

Users expect search systems that are fast (i.e., efficient) and return good results (i.e., effective), but these features are often in tension: effective in-depth content analysis can be slow, and fast systems often sacrifice quality. Building on a quarter of a century of experience developing techniques and building systems that connect users to relevant information, this proposal will lead to groundbreaking techniques for search and question answering (QA) that are effective as well as efficient. These capabilities will be demonstrated in high-impact applications on scientific texts such as the literature on coronaviruses or hydrology. Overall, my research will advance the scientific frontiers of both natural language processing (NLP) and information retrieval (IR). This proposal formulates search and question answering as ranking problems, and adopts an approach based on deep learning using a class of neural network architectures known as transformers. I will pursue the overall research vision from two complementary perspectives: (A) From the perspective of effectiveness, my research group will develop retrieval-specific self-supervision techniques and gain a better understanding of why transformers work as the basis for building improved ranking models. (B) From the perspective of efficiency, this research will build learned representations for ranking that are amenable to simple vector comparisons and develop model distillation techniques for accelerated inference. These separate threads will come together (C) in a principled framework for reasoning about effectiveness/efficiency tradeoffs, (D) demonstrated in applications that provide search and QA capabilities to literature in three scientific domains: biomedicine, hydrology, and artificial intelligence. Deployed prototypes will serve as a testbed for research and provide useful tools for stakeholders. Efforts along these lines have already begun: shortly after the start of the global COVID-19 pandemic, I led the development of Covidex (covidex.ai), an online and publicly accessible search engine for a collection of scientific articles related to coronaviruses. Such a system could be valuable, for example, to public health officials assessing the efficacy of different interventions and clinicians conducting meta-analyses. This work will achieve impact in four ways: (1) scientific innovations in the form of breakthroughs in deep learning methods that tackle search and QA, (2) computational artifacts in the form of open-source code, data, and models that will help foster adoption of the innovations arising from this research, (3) real-world applications for searching scientific literature in three domains, and (4) high-quality training opportunities. These efforts will contribute to the Canadian economy by enriching the ecosystem of ideas and talent around artificial intelligence and data science, complementing investments in these fields at the federal and provincial levels.