RI: Small: Domain-robust object detection through shape and context

计算机视觉在目标识别和检测方面取得了很大的进步，但当训练和部署时使用的数据非常不同时，性能会显著下降。这是有问题的，因为在许多情况下，在感兴趣的领域中的大样本集上重新训练模型可能是不可行的。例如，人工智能(AI)工具可以在一个国家或地区开发，使用该地区的培训数据，并出口到资源有限的地区，以收集新数据和重新训练模型。不幸的是，用户区域的视觉环境可能与开发者区域不同：印度的一些车辆看起来与美国的普通车辆不同；美国东海岸的房屋通常以砖为特色，但西海岸的房屋不太常见；环境因素(例如树叶和烟雾)可能会导致模型的表现不同。当计算机视觉系统在实践中应用时，对领域转换的健壮性对于可解释性和可信性是重要的。这个项目利用了这样的观察，即当这些对象在不同的领域(例如，照片和绘画)显示时，这些捕获对象的像素会发生变化，但对象的整体形状保持不变。此外，与感兴趣对象共同出现的对象集在域之间也相对一致。这个项目开发了新的视觉表示法，可以捕捉两个全局线索：形状和背景。虽然存在许多领域适应和泛化技术，但它们忽略了基于初步实验的可能对域转移更稳健的全局线索。第一种表示将中轴变换(MAT)调整为分层的、可学习的卷积表示。MAT计算对象的“骨架”，并使用密集的特征映射来开发表示，以确保有足够的信息供卷积网络捕获，以及建立对小移位的鲁棒性。其次，通过包含功能或语义相关的对象和环境提示(例如，共现文本或语音)的图来表示上下文，以提高模型识别新模式中的对象的能力。探索了用于使弱监督技术对域转移更健壮的技术，作为捕获非语义上下文的一种方式。然后，将这些全局表示与标准的基于外观的表示相结合，并通过领域泛化技术使其适应于新的领域或使其成为领域不变的。针对对象识别和检测，在包括照片真实感和艺术数据集、不同捕获条件以及可控移位场景(例如，模糊和掩蔽)的各种域移位场景中测试所得到的表示的域稳健性。将发布代码、任何人工创建的情况(数据)、如何为现有技术培训模型的明确协议以及详细的基准结果(定量和定性)，以确保重现性。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Computer vision has made great advancements in object recognition and detection, but performance drops significantly when the data used at training and deployment time are very different. This is problematic because in many situations, it may be infeasible to retrain the models on a large example set in the domain of interest. For example, artificial intelligence (AI) tools may be developed in one country or region, using that region’s training data, and exported to regions with limited resources to collect new data and retrain models. Unfortunately, the visual environment in the user region may be different from the developer region: some vehicles in India look different from common vehicles in the US; houses often feature bricks on the US East Coast but less frequently on the West Coast; environmental factors (e.g., foliage and smog) may cause models to behave differently. Being robust to domain shifts is important for interpretability and trust when computer vision systems are employed in practice. This project leverages the observation that while the pixels of captured objects change when these objects are shown in different domains (e.g., photographs vs paintings), the overall shape of the objects remains the same. Further, the set of objects that co-occur with the object of interest is also relatively consistent across domains. This project develops new visual representations that capture two global cues: shape and context. While numerous domain adaptations and generalization techniques exist, they have overlooked global cues that can potentially be more robust to domain shifts, based on preliminary experiments. The first proposed representation adapts the medial axis transform (MAT) into a hierarchical, learnable, convolutional representation. MAT computes the "skeleton" of an object, and a representation is developed using a dense feature map to ensure there is enough information for the convolutional network to capture, as well as to build robustness to small shifts. Second, context is represented through graphs containing functionally or semantically related objects, and ambient cues (such as co-occurring text or speech) to improve the model's ability to recognize objects in novel modalities. Techniques for making weakly-supervised techniques more robust to domain shifts are explored, as a way of capturing non-semantic context. Next, these global representations are combined with standard appearance-based ones and are adapted to novel domains or made domain-invariant through domain generalization techniques. The domain robustness of the resulting representations is tested in a variety of domain shift scenarios, including photorealistic and artistic datasets, different capture conditions, and controllable shift scenarios (e.g., blurring and masking), for both object recognition and detection. Code, any artificially created situations (data), clear protocols for how to train models for existing techniques, and detailed benchmarking results (quantitative and qualitative) will be released to ensure reproducibility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.