A Grammar-Based Approach to Visual-Haptic Object Perception

人们可以准确可靠地感知物体的形状，但这是如何发生的还不清楚。这种能力可能至少部分源于我们对视觉信息和触觉信息(当触摸或抓住物体时获得的信息)的使用。此外，如果我们学习根据视觉信息识别一个物体，我们通常可以在闭上眼睛但允许我们抓住它的情况下识别相同的物体。同样，如果我们学习根据触觉信息识别物体，我们通常可以在看到物体时识别出它，但不能触摸到它。换句话说，我们展示了物体形状信息的跨模式传输。眼睛和手的信息是如何在大脑中连接起来的，以产生物体形状的连贯表示？从这项研究中获得的见解可以帮助我们理解人类如何使用视觉和/或触觉来感知物体形状，并有助于开发在工业、医疗、军事和其他应用的多模式环境中运行的改进的机器人和其他人工智能系统。本项目发展了一种视觉-触觉物体形状感知理论，其中人们对物体相似的概念不是基于感觉特征，而是基于以抽象的、与形态无关的形式表示物体部分及其空间关系的潜在或隐藏变量。当查看对象、抓住对象或两者时，对象表示通过使用贝叶斯推理来推断基于语法的对象表示，从而使用概率“形状语法”来形式化对象表示。该模型是使用从人类视觉、触觉和视觉触觉物体形状感知的行为研究中获得的数据进行测试的。研究人员将探索从知觉新手向专家(例如放射科医生)转变的表征变化类型，并将评估知觉专业知识是否很好地表征为类别学习、语法学习、两者都不是。该研究计划还将开发一个大型公共代码数据库，用于重新创建复杂物体的视觉和触觉特征。这将允许其他研究人员使用3D打印机制造这些物体，增强研究地点之间的互补性和比较性。最后，在新兴的计算认知科学领域对本科生和研究生进行培训，将有助于培养跨越认知科学和计算机科学之间传统界限的新一代多学科科学家。

People can perceive the shape of objects accurately and reliably but how this occurs is not yet understood. This ability may stem, at least in part, from our use of both visual information and haptic information (information obtained when an object is touched or grasped). Moreover, if we learn to recognize an object based on visual information, we can often recognize the same object when our eyes are closed but we are allowed to grasp it. Similarly, if we learn to recognize an object based on haptic information, we can often recognize the object when we see it but cannot touch it. In other words, we exhibit cross-modal transfer of object shape information. How does information from the eyes and hands link up in the brain to yield a coherent representation of object shape? Insights obtained from this research can contribute both to our understanding of how humans perceive object shape using vision and/or touch and to development of improved robotic and other artificial intelligence systems operating in multi-modal settings in industrial, medical, military, and other applications.The present project develops a theory of visual-haptic object shape perception in which people's notions of object similarity are not based on sensory features but rather on latent or hidden variables that represent object parts and their spatial relations in an abstract, modality-independent format. Object representations are formalized using a probabilistic "shape grammar" with Bayesian inference used to infer grammar-based object representations when an object is viewed, when it is grasped, or both. The model is tested using data obtained from behavioral studies of visual, haptic, and visual-haptic object shape perception by humans. The investigators will explore the types of representational change that underlie the transition from perceptual novice to expert (e.g, radiologists) and will assess whether perceptual expertise is well characterized as category learning, grammar learning, both, or neither. The research program will also develop a large public database of code for re-creating both visual and haptic features of complex objects. This will allow other researchers to fabricate the objects using a 3D printer, enhancing complementarity and comparison across research sites. Finally, training undergraduate and graduate students in the emerging field of computational cognitive science will contribute to a new generation of multidisciplinary scientists working across traditional boundaries between cognitive science and computer science.