EAGER: Collaborative Research: Malleable Media to Support Interaction through Bi-Directional Touch Displays

与触摸屏的交互是引人注目的，因为在一定程度上，它复制了与我们环境中的对象的交互；您可以使用触摸屏在视觉上理解、接近、触摸和操纵对象。当然，这只是部分正确的，因为在最后一刻，你的手接触玻璃，其余的操作都是模拟的。对于许多应用程序，玻璃屏障是一种可以接受的折衷方案，因为屏幕上应用程序对您的操作(以视觉方式提供)的模拟响应足以弥合感知差距，欺骗您相信您实际上正在触摸按钮或图标；视觉反馈充当与对象缺失的触觉交互的代理。然而，在某些情况下，对于某些用户来说，玻璃屏障的存在会导致交互完全失败。对于盲人用户来说，尤其是对于他们来说，通过视觉关闭感知环不是一种选择，必须开发与计算中介环境交互的其他方式。这个探索性项目的主要目标是开发一种整页交互式触觉显示器，通过将触摸传感与其现有的微流体执行器相结合来创建整页响应表面，从而呈现PI所称的“可延展媒体”。这样的展示可以立即对它被触摸的方式做出切实的反应。通过超越屏幕的玻璃屏障，这种显示器将通过使按钮和图标等界面元素以及应用程序内容都可触摸来支持直接操作范例。因此，项目成果可能为盲人用户提供更多获取数字媒体的机会。但有视力的社区也将受益。在STEM领域工作的学生和专业人员都需要访问越来越多的高维数据，而这些数据无法通过语音轻松访问。像这项工作的焦点那样的交互式触觉显示将为盲人计算机用户和他们的有视力的同行提供这样的通道。这一探索性项目的智力价值来自于结合了PI现有研究的两个方面。第一项是他们最近在微流控触觉显示器方面的工作，第二项是他们将主动认知原理应用于人机界面设计的既定记录。主动认知假设了我们对环境的行动和我们对环境如何反应的感知之间的紧密耦合。有了这项新技术，PI就有了创建这样一个交互界面的方法，但目前还不知道这种交互应该是什么感觉。通过用手塑造平板电脑的表面来雕刻一个物体会是什么感觉？对于盲人用户来说，这将提供一种方法，不仅可以感受触觉对象，还可以创建这些对象。例如，这些对象可以用作表示复杂数据集中存在的模式或与数学模型交互的新方式。

Interaction with touchscreens is compelling because, in part, it replicates interaction with objects in our environment; you can use a touchscreen to visually apprehend, reach toward, touch and manipulate an object. Of course, this is only partly true because, at the last moment, your hand contacts glass and the rest of the manipulation is simulated. For many applications, the glass barrier is an acceptable compromise because the simulated response of the on-screen application to your action (delivered visually) is sufficient to close a perceptual gap, to fool you into believing that you are actually touching a button or icon; the visual feedback serves as proxy for the missing tactile interaction with the object. However, in some situations, and for some users, the presence of the glass barrier causes the interaction to fail completely. For blind users in particular, for whom closing a perceptual loop through vision is not an option, other means of interaction with computationally mediated environments must be developed. The primary objective of this exploratory project is to develop a full-page interactive tactile display that can render what the PIs term "malleable media" by combining touch sensing with their existing microfluidic actuators to create a full-page responsive surface. Such a display could provide an immediate tangible response in reaction to how it is being touched. By transcending the glass barrier of the screen, such a display would quite literally support a direct manipulation paradigm by making both interface elements, such as buttons and icons, and application content touchable. Thus, project outcomes are likely to provide greater accessibility to digital media by blind users. But the sighted community also stands to benefit. Both students and professionals working in STEM fields require access to increasingly high dimensional data that cannot be easily accessed via speech. An interactive tactile display such as that which is the focus of this work would provide such access, to blind computer users and their sighted counterparts alike.The intellectual merit of this exploratory project derives from combining two strands of existing research by the PIs. The first is their recent work on a microfluidic tactile display, while the second is their established record in applying principles of enactive cognition to the design of human-computer interfaces. Enactive cognition posits a tight coupling between our actions on the environment and our perception of how the environment is responding. With the new technology the PIs have the means of creating such an interactive surface, but it is not yet known what such interactions should "feel" like. What would it be like to sculpt an object by molding the surface of a tablet computer with one's hands? For blind users, this would provide a means for not just feeling tactile objects, but also for creating those objects. Such objects might be used as new ways of representing, for example, patterns present in complex data sets or of interacting with mathematical models.