CAREER: Natural Narratives and Multimodal Context as Weak Supervision for Learning Object Categories

该项目开发了一个框架来训练计算机视觉模型，用于从弱的、自然发生的语言监督（文本或语音）和其他多模态信号中检测物体。它考虑了动态设置，人类与他们的视觉环境互动，并参考遇到的物体，例如，“小心地把番茄植株放在地上”和“请放下电话，来摆桌子”，以及为人类观众写的字幕，以补充图像，例如，新闻文章的标题。使用这种基于语言的监督来训练检测系统的挑战在于，除了有用的信号外，语音中还包含许多不相关的标记。该项目将通过探索克服这一挑战的新途径，减少对昂贵且可能不自然的培训众包标签的需求，从而造福社会。它有可能使目标检测系统更具可扩展性，从而在各种设置中更广泛的用户群中更可用。开发的资源和工具将允许在不同的环境中进行自然的轻量级学习，例如，在不同的语言或图像类型中，已知的对象类别不起作用，或者像素以及人类引用对象的方式（不同的文化，医学，艺术）都发生了变化。这个项目为在体内而不是在体外学习提供了可能性；虽然这里的重点是对象类别，但多模态弱监督对于更广泛的任务是有用的。研究和教育通过当地社区外展和对来自不太知名大学的学生的研究指导，为学生培训的新项目，包括磨练研究生的写作技能，以及基于研究成果开发互动教育模块和演示，进行了整合。该项目创造性地连接了视觉与语言和对象检测两个领域，并开创了具有弱语言监督和大量潜在类词汇的对象检测模型的训练。语言通道中噪声的影响将通过三种互补的技术得到缓解，这三种技术分别是对单词的视觉具体性、文本在多大程度上涉及它所出现的视觉环境、以及学习到的弱监督模型是否在逻辑上是一致的。本文将使用两种互补的词域关联机制（度量学习和跨模态变压器），它们在弱监督检测中的应用是新颖的。重要的是，为了使检测可行，不仅要捕获图像-文本对的语义，还要捕获它们的话语关系。为了促进和消除词与物理环境的关联，后者将通过额外的模式来表示，即声音、运动、深度和触摸，这些模式要么存在于数据中，要么是估计的。该项目推进了多模态线索如何将图像和文本之间的关系语境化的知识；以前没有工作沿着多个通道（声音、深度、触摸、运动）建模图像-文本关系。最后，为了将对象的外观与这些对象的目的和使用联系起来，对象、属性和操作之间的关系将在语义上组织在一个图中，并且将提取用于表示涉及对象的活动的语法，仍然保持弱监督设置。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project develops a framework to train computer vision models for detection of objects from weak, naturally-occurring supervision of language (text or speech) and additional multimodal signals. It considers dynamic settings, where humans interact with their visual environment and refer to the encountered objects, e.g., “Carefully put the tomato plants in the ground” and “Please put the phone down and come set the table,” and captions written for a human audience to complement an image, e.g., news article captions. The challenge of using such language-based supervision for training detection systems is that along with useful signal, the speech contains many irrelevant tokens. The project will benefit society by exploring novel avenues for overcoming this challenge and reducing the need for expensive and potentially unnatural crowdsourced labels for training. It has the potential to make object detection systems more scalable and thus more usable by a broad user base in a variety of settings. The resources and tools developed would allow natural, lightweight learning in different environments, e.g., different languages or types of imagery where the well-known object categories are not useful or where there is a shift in both the pixels as well as the way in which humans refer to objects (different cultures, medicine, art). This project opens possibilities for learning in vivo rather than in vitro; while the focus here is on object categories, multimodal weak supervision is useful for a larger variety of tasks. Research and education are integrated through local community outreach and research mentoring for students from lesser-known universities, new programs for student training including honing graduate students' writing skills, and development of interactive educational modules and demos based on research findings. This project creatively connects two domains, vision-and-language, and object detection, and pioneers training of object detection models with weak language supervision and a large vocabulary of potential classes. The impact of noise in the language channel will be mitigated through three complementary techniques that model visual concreteness of words, to what extent the text refers to the visual environment it appears with, and whether the weakly-supervised models that are learned are logically consistent. Two complementary word-region association mechanisms will be used (metric learning and cross-modal transformers), whose application is novel for weakly-supervised detection. Importantly, to make detection feasible, not only the semantics of image-text pairs, but their discourse relationship, will be captured. To facilitate and disambiguate the association of words to a physical environment, the latter will be represented through additional modalities, namely sound, motion, depth and touch, which are either present in the data or estimated. This project advances knowledge of how multimodal cues contextualize the relation between image and text; no prior work has modeled image-text relationships along multiple channels (sound, depth, touch, motion). Finally, to connect the appearance of objects to the purpose and use of these objects, relationships between objects, properties and actions will be semantically organized in a graph, and grammars to represent activities involving objects will be extracted, still maintaining the weakly-supervised setting.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.