RI: Small: Collaborative Research: MatCam: A Camera that Sees Materials

该项目开发了第一个材质相机或MatCam，它可以输出物体材质的每像素标签及其属性，可用于视觉计算任务。在日常的真实的世界中，有大量的材料是有用的辨别，包括混凝土，金属，塑料，天鹅绒，缎子，沥青上的水层，地毯，瓷砖，木材和大理石。 用于识别材料的装置在开发新技术方面具有重要意义。例如，移动的机器人可以使用MatCam来确定地形是草地、砾石、路面还是雪地，以便优化机械控制。 在电子商务中，MatCam可以标记对象的材料成分，用于广告和库存。 在机器人、数字建筑、人机交互、智能车辆和先进制造等领域，潜在的应用是无限的。此外，材质图在几乎所有的视觉算法中都具有基础性的重要性，包括分割、特征匹配、场景识别、基于图像的渲染、基于上下文的搜索以及对象识别和运动估计。相机为更广泛的科学和工程社区带来了材料识别，就像深度相机目前在计算机视觉以外的许多领域中使用一样。这项研究将高准确度材料估计从实验室带入现实世界，以快速高准确度的每像素材料估计。 该计划有三个技术目标。 首先，利用探测机器人从多个角度观察表面，捕获并存储材料外观数据库。 这个大型的、结构化的和可操作的视觉数据集然后被用于开发计算外观模型。 一种新的方法，使用角反射率梯度集成的表面外观特征的表征。 使用训练数据和统计推断方法，这些模型被设计用于硬件实现。最终目标是将材质相机实现为一个近实时的傻瓜式材质采集原型，将RGB-D相机扩展到提供颜色、深度和材质的RGB-DM相机。 材料外观模型的硬件实现采用FPGA和SoC（片上系统）技术。

This project develops the first material camera, or MatCam, that outputs a per-pixel label of object material and its properties that can be used in visual computing tasks. In the everyday real world there are a vast number of materials that are useful to discern including concrete, metal, plastic, velvet, satin, water layer on asphalt, carpet, tile, wood, and marble. A device for identifying materials has important implications in developing new technologies. For example, a mobile robot may use a MatCam to determine whether the terrain is grass, gravel, pavement, or snow in order to optimize mechanical control. In e-commerce, the material composition of objects can be tagged by a MatCam for advertising and inventory. The potential applications are limitless in areas such as robotics, digital architecture, human-computer interaction, intelligent vehicles and advanced manufacturing. Furthermore, material maps have foundational importance in nearly all vision algorithms including segmentation, feature matching, scene recognition, image-based rendering, context-based search, and object recognition and motion estimation. The camera brings material recognition to the broader scientific and engineering communities, in a similar way that depth cameras are currently used in many fields outside of computer vision. This research brings high accuracy material estimation out of the lab and into the real-world for fast high-accuracy per-pixel material estimates. The program has three technical aims. First, a material appearance database is captured and stored with an exploration robot viewing surfaces from multiple angles. This large, structured and actionable visual dataset is then used to develop computational appearance models. A novel methodology using angular reflectance gradients is integrated for characterizing features of surface appearance. Using the training data and statistical inference methods, these models are designed for hardware implementation. The final aim is the material camera implementation as a near real-time prototype of point-and-shoot material acquisition that extends RGB-D cameras to RGB-DM cameras that provide color, depth, and material. The hardware implementation of the material appearance models utilizes FPGA and SoC (system-on-chip) technology.