RI: Small: Collaborative Research: MatCam: A Camera that Sees Materials

该项目开发了第一个材料相机，或MatCam，它输出对象材料及其属性的逐像素标签，可用于视觉计算任务。在日常的现实世界中，有大量的材料是有用的，包括混凝土，金属，塑料，天鹅绒，缎子，沥青上的水层，地毯，瓷砖，木材和大理石。一种用于识别材料的装置在开发新技术方面具有重要意义。例如，移动机器人可以使用MatCam来确定地形是草地、砾石、路面还是雪地，以优化机械控制。在电子商务中，可以通过MatCam标记物品的材料组成，用于广告和库存。在机器人、数字建筑、人机交互、智能汽车和先进制造等领域，潜在的应用是无限的。此外，材质贴图在几乎所有的视觉算法中都具有重要的基础作用，包括分割、特征匹配、场景识别、基于图像的渲染、基于上下文的搜索、物体识别和运动估计。该相机为更广泛的科学和工程界带来了材料识别，就像深度相机目前在计算机视觉以外的许多领域使用的方式一样。本研究将高精度材料估计从实验室带到现实世界，实现快速高精度逐像素材料估计。该计划有三个技术目标。首先，捕获并存储材料外观数据库，并用探索机器人从多个角度观察表面。然后，这个大型的、结构化的、可操作的可视化数据集被用于开发计算外观模型。提出了一种利用角反射率梯度表征表面形貌特征的新方法。利用训练数据和统计推理方法，设计了这些模型的硬件实现。最终目标是将材料相机实现为近实时的直拍材料采集原型，将RGB-D相机扩展到提供颜色、深度和材料的RGB-DM相机。材料外观模型的硬件实现采用FPGA和片上系统（SoC）技术。

This project develops the first material camera, or MatCam, that outputs a per-pixel label of object material and its properties that can be used in visual computing tasks. In the everyday real world there are a vast number of materials that are useful to discern including concrete, metal, plastic, velvet, satin, water layer on asphalt, carpet, tile, wood, and marble. A device for identifying materials has important implications in developing new technologies. For example, a mobile robot may use a MatCam to determine whether the terrain is grass, gravel, pavement, or snow in order to optimize mechanical control. In e-commerce, the material composition of objects can be tagged by a MatCam for advertising and inventory. The potential applications are limitless in areas such as robotics, digital architecture, human-computer interaction, intelligent vehicles and advanced manufacturing. Furthermore, material maps have foundational importance in nearly all vision algorithms including segmentation, feature matching, scene recognition, image-based rendering, context-based search, and object recognition and motion estimation. The camera brings material recognition to the broader scientific and engineering communities, in a similar way that depth cameras are currently used in many fields outside of computer vision. This research brings high accuracy material estimation out of the lab and into the real-world for fast high-accuracy per-pixel material estimates. The program has three technical aims. First, a material appearance database is captured and stored with an exploration robot viewing surfaces from multiple angles. This large, structured and actionable visual dataset is then used to develop computational appearance models. A novel methodology using angular reflectance gradients is integrated for characterizing features of surface appearance. Using the training data and statistical inference methods, these models are designed for hardware implementation. The final aim is the material camera implementation as a near real-time prototype of point-and-shoot material acquisition that extends RGB-D cameras to RGB-DM cameras that provide color, depth, and material. The hardware implementation of the material appearance models utilizes FPGA and SoC (system-on-chip) technology.