Pretrained Transformers for Effective and Efficient Information Access: BERT and Beyond

用户希望搜索系统快速(即高效)并返回良好的结果(即有效)，但这些功能经常处于紧张状态：有效的深度内容分析可能会很慢，而快速的系统往往会牺牲质量。基于25年来开发技术和建立连接用户与相关信息的系统的经验，这项提议将带来既有效又高效的搜索和问答(QA)的突破性技术。这些能力将在关于冠状病毒或水文学的文献等科学文本的高影响力应用中得到展示。总体而言，我的研究将推动自然语言处理(NLP)和信息检索(IR)的科学前沿。该方案将搜索和问题回答作为排序问题，并采用了一种基于深度学习的方法，该方法使用了一类称为转换器的神经网络结构。我将从两个互补的角度来追求总体研究愿景：(A)从有效性的角度来看，我的研究小组将开发特定于检索的自我监督技术，并更好地理解为什么变压器作为建立改进的排名模型的基础。(B)从效率的角度来看，这项研究将建立易于简单向量比较的学习排序表示法，并开发用于加速推理的模型蒸馏技术。这些独立的线索将(C)在一个关于有效性/效率权衡的原则性框架中结合在一起，(D)在为三个科学领域的文献提供搜索和QA功能的应用程序中演示：生物医学、水文学和人工智能。部署的原型将作为研究的试验台，并为利益相关者提供有用的工具。沿着这些方向的努力已经开始：在全球新冠肺炎大流行开始后不久，我领导开发了Covex.ai，这是一个在线和公众可访问的搜索引擎，可以收集与冠状病毒有关的科学文章。例如，这样的系统对于评估不同干预措施的有效性的公共卫生官员和进行荟萃分析的临床医生可能是有价值的。这项工作将在四个方面产生影响：(1)以深度学习方法突破的形式实现的科学创新，以解决搜索和质量保证问题；(2)以开放源代码、数据和模型的形式出现的计算制品，将有助于促进采用这项研究产生的创新；(3)在三个领域搜索科学文献的现实世界应用；以及(4)高质量的培训机会。这些努力将丰富围绕人工智能和数据科学的思想和人才生态系统，补充联邦和省级在这些领域的投资，从而为加拿大经济做出贡献。

Users expect search systems that are fast (i.e., efficient) and return good results (i.e., effective), but these features are often in tension: effective in-depth content analysis can be slow, and fast systems often sacrifice quality. Building on a quarter of a century of experience developing techniques and building systems that connect users to relevant information, this proposal will lead to groundbreaking techniques for search and question answering (QA) that are effective as well as efficient. These capabilities will be demonstrated in high-impact applications on scientific texts such as the literature on coronaviruses or hydrology. Overall, my research will advance the scientific frontiers of both natural language processing (NLP) and information retrieval (IR). This proposal formulates search and question answering as ranking problems, and adopts an approach based on deep learning using a class of neural network architectures known as transformers. I will pursue the overall research vision from two complementary perspectives: (A) From the perspective of effectiveness, my research group will develop retrieval-specific self-supervision techniques and gain a better understanding of why transformers work as the basis for building improved ranking models. (B) From the perspective of efficiency, this research will build learned representations for ranking that are amenable to simple vector comparisons and develop model distillation techniques for accelerated inference. These separate threads will come together (C) in a principled framework for reasoning about effectiveness/efficiency tradeoffs, (D) demonstrated in applications that provide search and QA capabilities to literature in three scientific domains: biomedicine, hydrology, and artificial intelligence. Deployed prototypes will serve as a testbed for research and provide useful tools for stakeholders. Efforts along these lines have already begun: shortly after the start of the global COVID-19 pandemic, I led the development of Covidex (covidex.ai), an online and publicly accessible search engine for a collection of scientific articles related to coronaviruses. Such a system could be valuable, for example, to public health officials assessing the efficacy of different interventions and clinicians conducting meta-analyses. This work will achieve impact in four ways: (1) scientific innovations in the form of breakthroughs in deep learning methods that tackle search and QA, (2) computational artifacts in the form of open-source code, data, and models that will help foster adoption of the innovations arising from this research, (3) real-world applications for searching scientific literature in three domains, and (4) high-quality training opportunities. These efforts will contribute to the Canadian economy by enriching the ecosystem of ideas and talent around artificial intelligence and data science, complementing investments in these fields at the federal and provincial levels.