RI: Small: Domain-robust object detection through shape and context

计算机视觉在目标识别和检测方面取得了很大的进步，但当训练和部署时使用的数据差异很大时，性能会显著下降。这是有问题的，因为在许多情况下，在感兴趣的领域的大样本集上重新训练模型可能是不可行的。例如，人工智能（AI）工具可能在一个国家或地区开发，使用该地区的训练数据，并导出到资源有限的地区，以收集新数据并重新训练模型。不幸的是，用户地区的视觉环境可能与开发者地区不同：印度的一些车辆看起来与美国的普通车辆不同；在美国东海岸，房屋通常以砖块为特色，但在西海岸则不那么常见；环境因素（例如，树叶和烟雾）可能导致模型表现不同。在实际应用中，对领域偏移的鲁棒性对计算机视觉系统的可解释性和信任度至关重要。这个项目利用了这样一种观察，即当这些物体在不同的领域（例如，照片与绘画）中显示时，所捕获的物体的像素会发生变化，但物体的整体形状保持不变。此外，与感兴趣的对象共同出现的对象集在各个领域也相对一致。这个项目开发了新的视觉表现，捕捉了两个全球线索：形状和环境。虽然存在许多领域适应和泛化技术，但根据初步实验，它们忽略了可能对领域转移更稳健的全局线索。第一种提出的表示将中间轴变换（MAT）转换成一种分层的、可学习的卷积表示。MAT计算对象的“骨架”，并使用密集的特征映射开发表示，以确保有足够的信息供卷积网络捕获，并建立对小位移的鲁棒性。其次，通过包含功能或语义相关对象和环境线索（如共同出现的文本或语音）的图形来表示上下文，以提高模型以新模式识别对象的能力。研究了使弱监督技术对领域转移更健壮的技术，作为捕获非语义上下文的一种方法。然后，将这些全局表示与标准的基于外观的表示结合起来，并适应新的领域或通过领域泛化技术使其成为领域不变性。结果表示的域鲁棒性在各种域移位场景中进行了测试，包括真实感和艺术数据集，不同的捕获条件和可控的移位场景（例如，模糊和掩蔽），用于对象识别和检测。代码，任何人为创建的情况（数据），关于如何训练现有技术模型的明确协议，以及详细的基准测试结果（定量和定性）将被发布，以确保再现性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computer vision has made great advancements in object recognition and detection, but performance drops significantly when the data used at training and deployment time are very different. This is problematic because in many situations, it may be infeasible to retrain the models on a large example set in the domain of interest. For example, artificial intelligence (AI) tools may be developed in one country or region, using that region’s training data, and exported to regions with limited resources to collect new data and retrain models. Unfortunately, the visual environment in the user region may be different from the developer region: some vehicles in India look different from common vehicles in the US; houses often feature bricks on the US East Coast but less frequently on the West Coast; environmental factors (e.g., foliage and smog) may cause models to behave differently. Being robust to domain shifts is important for interpretability and trust when computer vision systems are employed in practice. This project leverages the observation that while the pixels of captured objects change when these objects are shown in different domains (e.g., photographs vs paintings), the overall shape of the objects remains the same. Further, the set of objects that co-occur with the object of interest is also relatively consistent across domains. This project develops new visual representations that capture two global cues: shape and context. While numerous domain adaptations and generalization techniques exist, they have overlooked global cues that can potentially be more robust to domain shifts, based on preliminary experiments. The first proposed representation adapts the medial axis transform (MAT) into a hierarchical, learnable, convolutional representation. MAT computes the "skeleton" of an object, and a representation is developed using a dense feature map to ensure there is enough information for the convolutional network to capture, as well as to build robustness to small shifts. Second, context is represented through graphs containing functionally or semantically related objects, and ambient cues (such as co-occurring text or speech) to improve the model's ability to recognize objects in novel modalities. Techniques for making weakly-supervised techniques more robust to domain shifts are explored, as a way of capturing non-semantic context. Next, these global representations are combined with standard appearance-based ones and are adapted to novel domains or made domain-invariant through domain generalization techniques. The domain robustness of the resulting representations is tested in a variety of domain shift scenarios, including photorealistic and artistic datasets, different capture conditions, and controllable shift scenarios (e.g., blurring and masking), for both object recognition and detection. Code, any artificially created situations (data), clear protocols for how to train models for existing techniques, and detailed benchmarking results (quantitative and qualitative) will be released to ensure reproducibility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.