CHS: Small: Data-Driven Material Understanding and Decomposition

我们每天都接触到各种各样的材料(金属、木材、织物、花岗岩等)。这有助于我们理解世界。识别和模拟真实世界的材料长期以来一直是计算机视觉和图形学的核心挑战。最近，在大规模数据集上训练的深度学习模型的推动下，场景理解经历了研究活动的爆炸性增长。但人们的注意力主要集中在物体上；材料受到的关注较少，主要集中在实验室环境中的仔细测量上。当然，现实世界中的材料与这些实验室环境之间存在着很大的差距。PI的目标是弥合这一鸿沟，使人们能够“在野外”理解材料。为此，她的团队最近发布了大规模众包数据集(OpenSurFaces，Intrative Images in the Wild，Minc)，这些数据集已经在研究界得到了广泛的使用。利用这些数据开发新的材料分割和识别算法，PI的团队已经开发出最先进的方法，为数据驱动的材料理解开辟了新的可能性，这将影响室内设计、材料编辑、视觉搜索和机器人等广泛应用。项目成果(包括新的数据集、注释和代码)将完全公开和公开。PI积极指导康奈尔大学代表不足的少数族裔，并正在与康奈尔大学的女性计算(WICC)和编程女孩(GWC)合作，以接触初中生和高中生。这项研究将建立注释工具的原型，并将它们整合到康奈尔大学针对高中少数民族学生的暑期研讨会中。PI的小组还将组织材料理解竞赛(MUC)，以推动材料识别和分割以及内在图像分解方面的创新。该项目包括材料理解方面的两个主要技术推动力：1.用于材料理解的内在图像。本征图像分解的目的是将图像分解为材料、光照等内在属性。这种分解是不合适的，对野外的图像来说是具有挑战性的。这项工作将收集新的成对阴影和深度注释用于固有图像分解；引入新的感知度量来评估算法；解决联合材料识别和固有分解；以及开发概念验证应用程序，用于基于图像的编辑使用固有图像分解。用于语义理解的材料识别。在野外识别材料是极具挑战性的。这项工作将收集大规模的材料注释和点击数据，并训练弱监督识别算法；收集木材和金属等子类别的细粒材料数据；开发新的粗粒和细粒识别算法；以及开发用于智能材料搜索和形状材料分配的概念验证应用程序。

We are in daily contact with a rich range of materials (metals, woods, fabrics, granites, etc.) that contribute to how we understand the world. Recognizing and modeling real-world materials have long been core challenges in computer vision and graphics. Recently, scene understanding has experienced an explosion of research activity driven by deep learning models trained on large-scale datasets. But the focus has mainly been on objects; materials have received less attention, and it has predominantly focused on careful measurements in laboratory settings. Of course, there is a large gap between materials in the real world and these laboratory settings. The PI's goal is to bridge that gap, to enable material understanding "in the wild." Toward this end, her group recently released large-scale crowdsourced datasets (OpenSurfaces, Intrinsic Images in the Wild, MINC) that are already being used extensively in the research community. Using this data to develop new material segmentations and recognition algorithms, the PI's team has produced state-of-the-art methods which open up new possibilities for data-driven material understanding that will impact a wide range of applications such as interior design, material editing, visual search, and robotics. Project outcomes (including new datasets, annotations, and code) will be made fully open and public. The PI actively mentors underrepresented minorities at Cornell, and is working with Women in Computing at Cornell (WICC) and Girls Who Code (GWC) to reach middle and high school students. This research will build prototypes of the annotation tools, and integrate them into summer workshops at Cornell aimed at high school minority students. The PI's group will also organize a Material Understanding Competition (MUC) to drive innovation in material recognition and segmentation, and intrinsic image decomposition.This project includes two major technical thrusts in material understanding:1. Intrinsic images for material understanding. Intrinsic image decomposition aims to decompose images into intrinsic properties such as material and illumination. This decomposition is ill-posed and challenging for images in the wild. This work will collect new pairwise shading and depth annotations for intrinsic image decomposition; introduce a new perceptual metric to evaluate algorithms; solve for joint material recognition and intrinsic decomposition; and develop proof-of-concept applications for image-based editing using intrinsic image decomposition.2. Material recognition for semantic understanding. Recognizing materials in the wild is extremely challenging. This work will collect large-scale material annotations with "click" data and train weakly supervised recognition algorithms; collect fine-grained material data for subcategories like wood and metal; develop new algorithms for coarse and fine-grain recognition; and develop proof-of-concept applications for intelligent material search, and material assignment to shapes.