CAREER: Natural Narratives and Multimodal Context as Weak Supervision for Learning Object Categories

该项目开发了一个框架，用于训练计算机视觉模型，以从弱的、自然发生的语言（文本或语音）监督和其他多模态信号中检测对象。它考虑了动态设置，其中人类与他们的视觉环境进行交互并参考遇到的对象，例如，“小心地把番茄植物放在地上”和“请放下电话，过来摆桌子”，以及为人类观众编写的标题，以补充图像，例如，新闻标题将这种基于语言的监督用于训练检测系统的挑战在于，语音沿着有用的信号，包含许多不相关的标记。 该项目将通过探索克服这一挑战的新途径，减少对昂贵且可能不自然的众包标签的需求，从而造福社会。它有可能使目标检测系统更具可扩展性，从而在各种设置中更容易被广泛的用户群使用。开发的资源和工具将允许在不同的环境中进行自然的轻量级学习，例如，不同的语言或图像类型，其中众所周知的对象类别没有用，或者像素以及人类提及对象的方式都有变化（不同的文化，医学，艺术）。该项目为在体内而不是体外学习提供了可能性;虽然这里的重点是对象类别，但多模态弱监督对于更广泛的任务是有用的。研究和教育是通过当地社区的推广和研究辅导的学生从鲜为人知的大学，新的学生培训计划，包括磨练研究生的写作技巧，并根据研究结果的互动教育模块和演示的发展相结合。该项目创造性地将视觉和语言以及对象检测这两个领域联系起来，并开创了具有弱语言监督和大量潜在类词汇的对象检测模型的训练。语言通道中噪声的影响将通过三种互补技术来减轻，这些技术对单词的视觉具体性进行建模，文本在多大程度上涉及它出现的视觉环境，以及学习的弱监督模型是否在逻辑上一致。将使用两个互补的词区域关联机制（度量学习和交叉模态变换器），其应用对于弱监督检测是新颖的。重要的是，为了使检测可行，不仅要捕获图像-文本对的语义，还要捕获它们的话语关系。为了促进和消除单词与物理环境的关联，后者将通过额外的模态来表示，即声音，运动，深度和触摸，这些模态存在于数据中或估计。这个项目的知识如何多模态线索语境化图像和文本之间的关系，没有以前的工作已经建模图像-文本关系沿着多个通道（声音，深度，触摸，运动）。最后，为了将对象的外观与这些对象的目的和用途联系起来，对象、属性和动作之间的关系将在图中进行语义组织，并且将提取用于表示涉及对象的活动的语法，仍然保持着微弱的-该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的评估支持影响审查标准。

This project develops a framework to train computer vision models for detection of objects from weak, naturally-occurring supervision of language (text or speech) and additional multimodal signals. It considers dynamic settings, where humans interact with their visual environment and refer to the encountered objects, e.g., “Carefully put the tomato plants in the ground” and “Please put the phone down and come set the table,” and captions written for a human audience to complement an image, e.g., news article captions. The challenge of using such language-based supervision for training detection systems is that along with useful signal, the speech contains many irrelevant tokens. The project will benefit society by exploring novel avenues for overcoming this challenge and reducing the need for expensive and potentially unnatural crowdsourced labels for training. It has the potential to make object detection systems more scalable and thus more usable by a broad user base in a variety of settings. The resources and tools developed would allow natural, lightweight learning in different environments, e.g., different languages or types of imagery where the well-known object categories are not useful or where there is a shift in both the pixels as well as the way in which humans refer to objects (different cultures, medicine, art). This project opens possibilities for learning in vivo rather than in vitro; while the focus here is on object categories, multimodal weak supervision is useful for a larger variety of tasks. Research and education are integrated through local community outreach and research mentoring for students from lesser-known universities, new programs for student training including honing graduate students' writing skills, and development of interactive educational modules and demos based on research findings. This project creatively connects two domains, vision-and-language, and object detection, and pioneers training of object detection models with weak language supervision and a large vocabulary of potential classes. The impact of noise in the language channel will be mitigated through three complementary techniques that model visual concreteness of words, to what extent the text refers to the visual environment it appears with, and whether the weakly-supervised models that are learned are logically consistent. Two complementary word-region association mechanisms will be used (metric learning and cross-modal transformers), whose application is novel for weakly-supervised detection. Importantly, to make detection feasible, not only the semantics of image-text pairs, but their discourse relationship, will be captured. To facilitate and disambiguate the association of words to a physical environment, the latter will be represented through additional modalities, namely sound, motion, depth and touch, which are either present in the data or estimated. This project advances knowledge of how multimodal cues contextualize the relation between image and text; no prior work has modeled image-text relationships along multiple channels (sound, depth, touch, motion). Finally, to connect the appearance of objects to the purpose and use of these objects, relationships between objects, properties and actions will be semantically organized in a graph, and grammars to represent activities involving objects will be extracted, still maintaining the weakly-supervised setting.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.