Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials
Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials
复制标题
DOI：
10.1109/cvpr.2017.478
复制
发表时间：
2017-04
期刊：
影响因子：
--
通讯作者：
Wenzhen Yuan;Shaoxiong Wang;Siyuan Dong;E. Adelson
中科院分区：
文献类型：
--
作者：
Wenzhen Yuan;Shaoxiong Wang;Siyuan Dong;E. Adelson
为了让机器与物理世界进行交互，它们必须了解它们遇到的物体和材料的物理特性。我们使用织物作为具有丰富机械性能的可变形材料的示例。薄的柔性织物，当悬垂时，往往看起来不同于重的硬织物。摸起来也不一样。使用118个织物样本的集合，我们捕获了悬垂织物的颜色和深度图像沿着来自高分辨率触摸传感器的触觉数据。然后，我们试图通过联合训练三种模式的CNN来关联视觉和触觉的信息。通过CNN，每个输入，无论模态如何，都会生成一个记录织物物理特性的嵌入向量。通过比较嵌入向量，我们的系统能够查看织物图像并预测它的感觉，反之亦然。我们还表明，在视觉和触摸数据上联合训练的系统在纯粹使用视觉输入进行测试时，可以优于仅在视觉数据上训练的类似系统。
For machines to interact with the physical world, they must understand the physical properties of objects and materials they encounter. We use fabrics as an example of a deformable material with a rich set of mechanical properties. A thin flexible fabric, when draped, tends to look different from a heavy stiff fabric. It also feels different when touched. Using a collection of 118 fabric samples, we captured color and depth images of draped fabrics along with tactile data from a high-resolution touch sensor. We then sought to associate the information from vision and touch by jointly training CNNs across the three modalities. Through the CNN, each input, regardless of the modality, generates an embedding vector that records the fabrics physical property. By comparing the embedding vectors, our system is able to look at a fabric image and predict how it will feel, and vice versa. We also show that a system jointly trained on vision and touch data can outperform a similar system trained only on visual data when tested purely with visual inputs.