CAREER: Learning Structured Models with Natural Language Supervision

目前的机器学习模型很难理解视觉场景、执行家务，以及完成其他需要将低级感知和动作与高级常识和背景知识相结合的任务。这个职业项目将使用语言来弥合这一差距，开发使用基于语言的数据集注释和大型文本语料库的技术来指导机器人学、计算机视觉和其他问题领域的机器学习模型的培训。在自然语言监督下学习的新方法将减少训练机器学习模型所需的数据量，并使最终用户能够在没有复杂的正式规范的情况下塑造模型行为。该项目将为本科生和研究生提供研究培训，并将纳入一个新的讲习班系列，将学术语言处理研究人员和其他应用领域的研究人员联系起来(重点是为历史上被边缘化群体的学生提供学习和社区建设机会)。该项目的教育部分将开发关于人工智能系统中自然语言处理和人为因素的新课程材料，目标是高中生和本科生以及研究自动化决策系统影响的非技术性行业团体(如记者和政策研究人员)。该项目的技术核心是一系列新的概率潜变量模型，其中对计划或感知的潜在表征联合生成任务数据和自然语言注释。当有语言注释时，它们可以直接监督这些潜在表征的内容；在没有注释的例子中，来自文本语料库的信息可以用来约束潜在表征的分布。因此，语言扮演着两个角色：一个是关于个别训练实例结构的信息来源，另一个是一般任务级背景知识的来源。研究将产生这一建模框架的具体实例，用于政策学习、语言建模和场景理解，使用语言生成结构化、可组合的模型，将深度学习工具包的灵活性与符号表示的样本效率和可控性相结合，同时既不需要大量标记的数据集，也不需要精确形式化的符号域。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Current machine learning models struggle to understand visual scenes, perform household chores, and complete other tasks that require integrating low-level perception and action with high-level common-sense and background knowledge. This CAREER project will use language to bridge this gap by developing techniques that use language-based dataset annotations and large text corpora to guide training of machine learning models for robotics, computer vision, and other problem domains. New approaches for learning with natural language supervision will reduce the amount of data needed to train machine learning models and enable end users to shape model behavior without complex formal specifications. The project will provide research training to undergraduate and graduate students, and will be integrated into a new workshop series that connects academic language processing researchers and researchers in other application areas (with a focus on providing learning and community-building opportunities for students from historically marginalized groups). The educational component of the project will develop new curriculum materials on natural language processing and human factors in artificial intelligence systems, targeting high school and undergraduate students as well as non-technical industry groups (like journalists and policy researchers) studying the effects of automated decision-making systems.The technical core of this project is a new family of probabilistic latent variable models in which latent representations of plans or percepts jointly generate task data and natural language annotations. When language annotations are available, they can directly supervise the content of these latent representations; on unannotated examples, information from text corpora may be used to constrain latent representations' distribution. Language thus plays two roles: as a source of information about the structure of individual training examples and a source of general, task-level background knowledge. Research will yield concrete instantiations of this modeling framework for policy learning, language modeling, and scene understanding, using language to produce structured, composable models that combine the flexibility of the deep learning toolkit with the sample efficiency and controllability of symbolic representations, while requiring neither massive labeled datasets nor precisely formalized symbolic domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.