CAREER: Learning Structured Models with Natural Language Supervision

当前的机器学习模型很难理解视觉场景，执行家务，并完成其他需要将低级感知和行动与高级常识和背景知识相结合的任务。这个CAREER项目将使用语言来弥合这一差距，开发使用基于语言的数据集注释和大型文本语料库的技术，以指导机器人，计算机视觉和其他问题领域的机器学习模型的训练。使用自然语言监督进行学习的新方法将减少训练机器学习模型所需的数据量，并使最终用户能够在没有复杂的正式规范的情况下塑造模型行为。该项目将为本科生和研究生提供研究培训，并将纳入一个新的研讨会系列，将学术语言处理研究人员和其他应用领域的研究人员联系起来（重点是为历史上边缘化群体的学生提供学习和社区建设机会）。该项目的教育部分将开发关于自然语言处理和人工智能系统中的人为因素的新课程材料，针对高中生和本科生以及非技术行业群体（如记者和政策研究人员）研究自动化决策的影响-该项目的技术核心是一个新的概率潜变量模型家族，其中计划或感知的潜在表示联合生成任务数据和自然语言注释。当语言注释可用时，它们可以直接监督这些潜在表征的内容;在未注释的示例中，来自文本语料库的信息可以用于约束潜在表征的分布。语言扮演着两个角色：作为关于个体训练示例的结构的信息源和一般任务级背景知识的源。研究将产生该建模框架的具体实例，用于政策学习、语言建模和场景理解，使用语言生成结构化、可组合的模型，将深度学习工具包的灵活性与符号表示的样本效率和可控性结合起来，同时既不需要大量的标记数据集，也不需要精确形式化的符号域。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，认为值得支持。

Current machine learning models struggle to understand visual scenes, perform household chores, and complete other tasks that require integrating low-level perception and action with high-level common-sense and background knowledge. This CAREER project will use language to bridge this gap by developing techniques that use language-based dataset annotations and large text corpora to guide training of machine learning models for robotics, computer vision, and other problem domains. New approaches for learning with natural language supervision will reduce the amount of data needed to train machine learning models and enable end users to shape model behavior without complex formal specifications. The project will provide research training to undergraduate and graduate students, and will be integrated into a new workshop series that connects academic language processing researchers and researchers in other application areas (with a focus on providing learning and community-building opportunities for students from historically marginalized groups). The educational component of the project will develop new curriculum materials on natural language processing and human factors in artificial intelligence systems, targeting high school and undergraduate students as well as non-technical industry groups (like journalists and policy researchers) studying the effects of automated decision-making systems.The technical core of this project is a new family of probabilistic latent variable models in which latent representations of plans or percepts jointly generate task data and natural language annotations. When language annotations are available, they can directly supervise the content of these latent representations; on unannotated examples, information from text corpora may be used to constrain latent representations' distribution. Language thus plays two roles: as a source of information about the structure of individual training examples and a source of general, task-level background knowledge. Research will yield concrete instantiations of this modeling framework for policy learning, language modeling, and scene understanding, using language to produce structured, composable models that combine the flexibility of the deep learning toolkit with the sample efficiency and controllability of symbolic representations, while requiring neither massive labeled datasets nor precisely formalized symbolic domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.