CHS: Small: Data-Driven Material Understanding and Decomposition

我们每天都在接触丰富的材料（金属、木材、织物、花岗岩等），这些材料有助于我们如何理解世界。识别和建模真实世界的材料一直是计算机视觉和图形学的核心挑战。最近，在大规模数据集训练的深度学习模型的推动下，场景理解的研究活动呈爆炸式增长。但焦点主要集中在物体上；材料受到的关注较少，它主要集中在实验室环境中的仔细测量。当然，现实世界中的材料与这些实验室设置之间存在很大差距。PI的目标是弥合这一差距，使人们能够“在野外”理解材料。为此，她的团队最近发布了大规模的众包数据集（OpenSurfaces, Intrinsic Images in the Wild， MINC），这些数据集已经在研究界广泛使用。PI的团队利用这些数据开发新的材料分割和识别算法，创造了最先进的方法，为数据驱动的材料理解开辟了新的可能性，这将影响广泛的应用，如室内设计、材料编辑、视觉搜索和机器人技术。项目成果（包括新的数据集、注释和代码）将完全公开。PI积极指导康奈尔大学未被充分代表的少数族裔，并与康奈尔大学计算机女性协会（Women in Computing at Cornell， WICC）和编程女孩协会（Girls Who Code， GWC）合作，以接触初高中学生。这项研究将建立注释工具的原型，并将其整合到康奈尔大学针对高中少数民族学生的暑期研讨会中。PI小组还将组织一场材料理解竞赛（MUC），以推动材料识别和分割以及内在图像分解方面的创新。本项目在材料理解方面包括两个主要的技术重点：1。材料理解的内在图像。内在图像分解的目的是将图像分解为材料、光照等内在属性。这种分解是病态的，对野外的图像具有挑战性。这项工作将收集新的两两阴影和深度注释，用于图像的内在分解；引入一种新的感知度量来评估算法；求解节理材料识别和本征分解；并使用内在图像分解开发基于图像编辑的概念验证应用程序。语义理解的材料识别。在野外识别材料是极具挑战性的。这项工作将收集带有“点击”数据的大规模材料注释，并训练弱监督识别算法；收集细粒度材料的子类别数据，如木材和金属；开发新的粗粒和细粒识别算法；并开发概念验证应用程序，用于智能材料搜索和材料形状分配。

We are in daily contact with a rich range of materials (metals, woods, fabrics, granites, etc.) that contribute to how we understand the world. Recognizing and modeling real-world materials have long been core challenges in computer vision and graphics. Recently, scene understanding has experienced an explosion of research activity driven by deep learning models trained on large-scale datasets. But the focus has mainly been on objects; materials have received less attention, and it has predominantly focused on careful measurements in laboratory settings. Of course, there is a large gap between materials in the real world and these laboratory settings. The PI's goal is to bridge that gap, to enable material understanding "in the wild." Toward this end, her group recently released large-scale crowdsourced datasets (OpenSurfaces, Intrinsic Images in the Wild, MINC) that are already being used extensively in the research community. Using this data to develop new material segmentations and recognition algorithms, the PI's team has produced state-of-the-art methods which open up new possibilities for data-driven material understanding that will impact a wide range of applications such as interior design, material editing, visual search, and robotics. Project outcomes (including new datasets, annotations, and code) will be made fully open and public. The PI actively mentors underrepresented minorities at Cornell, and is working with Women in Computing at Cornell (WICC) and Girls Who Code (GWC) to reach middle and high school students. This research will build prototypes of the annotation tools, and integrate them into summer workshops at Cornell aimed at high school minority students. The PI's group will also organize a Material Understanding Competition (MUC) to drive innovation in material recognition and segmentation, and intrinsic image decomposition.This project includes two major technical thrusts in material understanding:1. Intrinsic images for material understanding. Intrinsic image decomposition aims to decompose images into intrinsic properties such as material and illumination. This decomposition is ill-posed and challenging for images in the wild. This work will collect new pairwise shading and depth annotations for intrinsic image decomposition; introduce a new perceptual metric to evaluate algorithms; solve for joint material recognition and intrinsic decomposition; and develop proof-of-concept applications for image-based editing using intrinsic image decomposition.2. Material recognition for semantic understanding. Recognizing materials in the wild is extremely challenging. This work will collect large-scale material annotations with "click" data and train weakly supervised recognition algorithms; collect fine-grained material data for subcategories like wood and metal; develop new algorithms for coarse and fine-grain recognition; and develop proof-of-concept applications for intelligent material search, and material assignment to shapes.