RI: Small: Multilingual Supervision for Object Detection under Geographic Domain and Concept Shifts

RI：小型：地理领域和概念转变下目标检测的多语言监督

• 批准号: 2329992
• 负责人: Adriana Kovashka
• 金额: $ 58.8万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2329992&HistoricalAwards=false
• 关键词: RI; Small; Multilingual; Supervision; Object

Object detection methods are trained with data that is geographically biased in terms of both the visual appearance and supervision. This limits applicability to populations outside North America and Europe. The recent trend of training object detection systems with information from text accompanying images, also implies a shift in the supervision these systems receive. Members of different countries/cultures may produce text with varying structure and content for the same image. This project studies how objects (e.g., cleaning equipment) appear differently in countries with different geographic, cultural, and economic conditions, and how information about these countries may be used to bridge the gaps in appearance. The project also investigates which objects the speakers of a particular language mention and at what level of specificity (e.g., “dog” vs “beagle”). Finally, the research studies how to best enable detection of rare, culture-specific concepts. This project involves extensive work with members of different countries/cultures who will consult the researchers on reasons for varying object appearance and naming and discuss the benefits and dangers of computer vision in their everyday lives. It will also support training for graduate and undergraduate students, from a diverse population in the Pittsburgh community.This project examines the reasons for objects to appear differently in different domains (countries) and develops techniques to enable detection in target domains with limited data (e.g., Africa) using the knowledge from language models, such as country-specific distinctive features of objects and information about the visual environment in a country and combining these with domain adaptation and prompt learning. Very limited prior work studied the problem of visual appearance and background shifts across geographic regions. This shows that standard domain adaptation methods do not help to bridge domain gaps; this project instead seeks help from diverse types of knowledge in language models. The project also studies how cultures provide different descriptions (captions) of images, in terms of objects mentioned and the levels of an entity hierarchy used to name objects. Prior work in language-supervised object detection fails to account for cross-language differences in captions, and multilingual work in retrieval does not focus on objects. The researchers will develop techniques to leverage machine translation in a manner that is aware of different specificity of object naming, and to share information across languages using soft positives in a contrastive learning framework. Furthermore, the project will study how to enable training for objects specific to given cultures, that may not be available in the vision-language pretraining set. While open-vocabulary work examines detection of rare objects, it does not exclude those from the pre-training set. This project aims to reduce a model’s reliance on object names, boost reliance on descriptive context (e.g., attributes), and leverage related categories. Beyond the project’s findings, the team will develop a tool and collect new annotations to share with the community: bounding boxes for datasets that only offer captions in different languages, and captions for datasets collected in different countries, but only providing object annotations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对象检测方法使用在视觉外观和监督方面具有地理偏见的数据进行训练。这限制了对北美和欧洲以外人群的适用性。最近的趋势是用来自伴随图像的文本的信息来训练对象检测系统，这也意味着这些系统所接受的监督的转变。不同国家/文化的成员可能会为同一图像生成具有不同结构和内容的文本。该项目研究对象（例如，清洁设备）在具有不同地理、文化和经济条件的国家中出现不同，以及如何使用关于这些国家的信息来弥合外观上的差距。该项目还调查了特定语言的使用者提到了哪些对象，以及具体程度如何（例如，“狗”vs“小猎犬”）。最后，研究如何最好地使罕见的，文化特定的概念检测。该项目涉及与不同国家/文化的成员进行广泛的工作，他们将向研究人员咨询不同物体外观和命名的原因，并讨论计算机视觉在日常生活中的好处和危险。该项目还将支持对匹兹堡社区不同人群的研究生和本科生进行培训。该项目研究了对象在不同领域（国家）出现不同的原因，并开发了能够在数据有限的目标领域进行检测的技术（例如，非洲）使用来自语言模型的知识，如特定国家的物体的独特特征和关于一个国家视觉环境的信息，并将这些与领域适应和快速学习相结合。非常有限的先前工作研究的问题，视觉外观和背景的地理区域之间的变化。这表明，标准的领域适应方法无助于弥合领域差距;该项目反而寻求语言模型中各种类型知识的帮助。该项目还研究了文化如何提供不同的描述（标题）的图像，在提到的对象和用于命名对象的实体层次结构的水平。语言监督对象检测的先前工作未能考虑字幕中的跨语言差异，并且检索中的多语言工作并不关注对象。研究人员将开发技术，以了解对象命名的不同特异性的方式利用机器翻译，并在对比学习框架中使用软阳性来跨语言共享信息。此外，该项目还将研究如何为特定于特定文化的对象进行训练，这些对象可能在视觉语言预训练集中不可用。虽然开放词汇表工作检查了稀有对象的检测，但它并不排除预训练集中的对象。该项目旨在减少模型对对象名称的依赖，提高对描述性上下文的依赖（例如，属性），并利用相关类别。除了该项目的发现，该团队将开发一个工具，并收集新的注释与社区分享：仅提供不同语言字幕的数据集的边界框，以及在不同国家收集的数据集的字幕，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响进行评估，被认为值得支持审查标准。