RI: Small: Collaborative Research: Seeing Surfaces: Actionable Surface Properties from Vision

该项目旨在使计算机和机器人能够根据其外观(外观)来估计表面(感觉)的可操作的物理属性。其关键思想是利用辐射测量和物理表面特征之间的深度相互交织的关系。通过学习模型，明确在辐射外观属性的全部和部分测量中编码的物理表面属性，计算机可以使用新型相机系统从被动观测中估计现实世界表面的关键物理属性。该项目为将这些模型和估计算法集成到场景理解、机器人行动规划和高效视觉传感中铺平了道路。研究结果为计算机视觉提供了一种目前缺失但基本的能力，使计算机视觉、机器人和计算机图形学领域的许多应用受益。该项目为本科生和研究生提供了实践研究的机会，并被整合到德雷克塞尔和罗格斯大学教授的PI的本科生和研究生课程中。它们也被用作K-12外展活动的背景，包括高中和初中导师计划。数据收集活动为K-12学生接触物理和计算机科学提供了一个理想的平台。本研究探索了从图像和详细的表面反射率测量推断可操作的表面属性的方法。研究活动集中于四个具体目标：1)可操作的和非控制的大规模数据收集日常表面的可操作的物理属性和外观测量，2)推导用于从局部表面外观推断物理属性的预测模型，3)集成包括对象和场景信息的全局语义上下文，以及4)开发有效的外观捕获并将其用于从外观感知新的物理。这些研究成果共同回答了一个基本问题，即计算机视觉如何在不接触表面和知道表面是什么的情况下预测表面的物理属性，为计算机视觉-行动奠定了基础。

This project is to enable computers and robots the capability of estimating actionable, physical properties of surfaces (the feel) from their appearance (the looks). The key idea is to leverage the deeply interwoven relation between radiometric and physical surface characteristics. By learning models that make explicit the physical surface properties encoded in full and partial measurements of radiometric appearance properties, computers can estimate crucial physical properties of real-world surfaces from passive observations with novel camera systems. This project paves the path for integrating these models and estimation algorithms into scene understanding, robotic action planning, and efficient visual sensing. The research results provide a currently missing but fundamental capability to computer vision that benefits a number of applications in areas of computer vision, robotics, and computer graphics. The project provides hands-on research opportunities for both undergraduate and graduate students and are integrated in the PIs' undergraduate and graduate courses taught at Drexel and Rutgers. They are also used as a backdrop for K-12 outreach activities including high school and middle school mentorship programs. The data collection activities provide an ideal platform to expose K-12 students to physics and computer science.This research investigates the methods to infer actionable surface properties from images and detailed surface reflectance measurements. The research activities are centered on four specific aims: 1) controlled and uncontrolled large-scale data collection of actionable physical properties and appearance measurements of everyday surfaces, 2) derivation of prediction models for deducing physical properties from local surface appearance, 3) integration of global semantic context including object and scene information, and 4) development of efficient appearance capture and its use for novel physics-from-appearance sensing. These research thrusts collectively answer the fundamental question of how computer vision can anticipate the physical properties of a surface without touching it and knowing what it is, laying the foundation for computational vision-for-action.