RI: Small: Multilingual Supervision for Object Detection under Geographic Domain and Concept Shifts

目标检测方法使用在视觉外观和监督方面都存在地理偏差的数据进行训练。这限制了北美和欧洲以外的人口的适用性。最近的趋势是利用伴随图像的文本信息来训练目标检测系统，这也意味着这些系统接受的监督的转变。不同国家/文化的成员可能会为同一图像制作具有不同结构和内容的文本。该项目研究不同地理、文化和经济条件的国家的物品(如清洁设备)的外观有何不同，以及如何利用关于这些国家的信息来弥合外观上的差距。该项目还调查了说某种语言的人提到了哪些对象以及具体程度(例如，“狗”与“小猎犬”)。最后，这项研究研究了如何最好地检测罕见的、特定于文化的概念。这个项目涉及与不同国家/文化的成员进行广泛的工作，他们将就不同物体外观和命名的原因咨询研究人员，并讨论计算机视觉在日常生活中的好处和危险。该项目还将支持对匹兹堡社区不同人群的研究生和本科生的培训。该项目研究物体在不同领域(国家)出现不同的原因，并开发技术，利用来自语言模型的知识，如物体特定于国家的独特特征和关于一个国家视觉环境的信息，在数据有限的目标领域(如非洲)进行检测，并将这些与领域适应和快速学习相结合。以前的工作非常有限，研究了视觉外观和背景跨地理区域移动的问题。这表明标准的领域适配方法无助于弥合领域差距；相反，该项目寻求语言模型中不同类型的知识的帮助。该项目还研究了文化如何根据所提到的对象和用于命名对象的实体层次结构的级别，对图像提供不同的描述(标题)。以往的语言监督对象检测工作不能考虑字幕中的跨语言差异，检索中的多语言工作也不关注对象。研究人员将开发技术，以意识到对象命名的不同特殊性来利用机器翻译，并在对比学习框架中使用软肯定词跨语言共享信息。此外，该项目还将研究如何对特定文化中的特定对象进行培训，这些对象可能无法在视觉语言预训集中找到。虽然开放词汇表作业检查对稀有物体的检测，但它并不排除那些预先训练的对象。该项目旨在减少模型对对象名称的依赖，增加对描述性上下文(例如，属性)的依赖，并利用相关类别。除了该项目的发现之外，该团队还将开发一个工具并收集新的注释与社区共享：只提供不同语言标题的数据集的边界框，以及在不同国家收集的数据集的标题，但只提供对象注释。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Object detection methods are trained with data that is geographically biased in terms of both the visual appearance and supervision. This limits applicability to populations outside North America and Europe. The recent trend of training object detection systems with information from text accompanying images, also implies a shift in the supervision these systems receive. Members of different countries/cultures may produce text with varying structure and content for the same image. This project studies how objects (e.g., cleaning equipment) appear differently in countries with different geographic, cultural, and economic conditions, and how information about these countries may be used to bridge the gaps in appearance. The project also investigates which objects the speakers of a particular language mention and at what level of specificity (e.g., “dog” vs “beagle”). Finally, the research studies how to best enable detection of rare, culture-specific concepts. This project involves extensive work with members of different countries/cultures who will consult the researchers on reasons for varying object appearance and naming and discuss the benefits and dangers of computer vision in their everyday lives. It will also support training for graduate and undergraduate students, from a diverse population in the Pittsburgh community.This project examines the reasons for objects to appear differently in different domains (countries) and develops techniques to enable detection in target domains with limited data (e.g., Africa) using the knowledge from language models, such as country-specific distinctive features of objects and information about the visual environment in a country and combining these with domain adaptation and prompt learning. Very limited prior work studied the problem of visual appearance and background shifts across geographic regions. This shows that standard domain adaptation methods do not help to bridge domain gaps; this project instead seeks help from diverse types of knowledge in language models. The project also studies how cultures provide different descriptions (captions) of images, in terms of objects mentioned and the levels of an entity hierarchy used to name objects. Prior work in language-supervised object detection fails to account for cross-language differences in captions, and multilingual work in retrieval does not focus on objects. The researchers will develop techniques to leverage machine translation in a manner that is aware of different specificity of object naming, and to share information across languages using soft positives in a contrastive learning framework. Furthermore, the project will study how to enable training for objects specific to given cultures, that may not be available in the vision-language pretraining set. While open-vocabulary work examines detection of rare objects, it does not exclude those from the pre-training set. This project aims to reduce a model’s reliance on object names, boost reliance on descriptive context (e.g., attributes), and leverage related categories. Beyond the project’s findings, the team will develop a tool and collect new annotations to share with the community: bounding boxes for datasets that only offer captions in different languages, and captions for datasets collected in different countries, but only providing object annotations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.