EAGER: Collaborative Research: Malleable Media to Support Interaction through Bi-Directional Touch Displays

触屏互动之所以引人注目，部分原因在于它复制了我们与环境中物体的互动；你可以用触摸屏直观地理解、触摸和操纵一个物体。当然，这只是部分正确，因为在最后一刻，你的手接触玻璃，其余的操作是模拟的。对于许多应用来说，玻璃屏障是一个可以接受的折衷方案，因为屏幕上的应用程序对你的动作的模拟反应（视觉上的）足以消除感知上的差距，让你误以为你真的触摸到了按钮或图标；视觉反馈代替了缺失的与物体的触觉交互。然而，在某些情况下，对于某些用户来说，玻璃屏障的存在会导致交互完全失败。特别是对于盲人用户来说，通过视觉关闭感知循环是不可能的，因此必须开发与计算媒介环境交互的其他手段。这个探索性项目的主要目标是开发一种全页交互式触觉显示器，通过将触摸传感与现有的微流体执行器相结合，可以呈现pi所称的“可塑介质”，从而创建一个全页响应表面。这样的显示器可以对它被触摸的方式做出即时的、切实的反应。通过超越屏幕的玻璃屏障，这样的显示将通过使界面元素（如按钮和图标）和应用程序内容都可触摸，从字面上支持直接操作范例。因此，项目成果很可能使盲人用户更容易获得数字媒体。但视力正常的群体也会从中受益。在STEM领域工作的学生和专业人员都需要访问越来越高维的数据，而这些数据无法通过语音轻松访问。一种互动式触觉显示器，就像这项工作的重点一样，将为盲人和视力正常的电脑用户提供这样的途径。这个探索性项目的智力价值来自于pi现有研究的两股结合。第一个是他们最近在微流控触觉显示器上的工作，而第二个是他们在将动作认知原理应用于人机界面设计方面的既定记录。动作认知假设我们对环境的行为和我们对环境如何反应的感知之间存在紧密耦合。有了这项新技术，pi就有办法创建这样一个交互界面，但目前还不知道这种交互应该是什么感觉。用手塑造平板电脑的表面会是什么样子？对于盲人用户来说，这将提供一种方法，不仅可以感知触觉对象，还可以创建这些对象。这些对象可能被用作表示（例如）复杂数据集中出现的模式或与数学模型交互的新方法。

Interaction with touchscreens is compelling because, in part, it replicates interaction with objects in our environment; you can use a touchscreen to visually apprehend, reach toward, touch and manipulate an object. Of course, this is only partly true because, at the last moment, your hand contacts glass and the rest of the manipulation is simulated. For many applications, the glass barrier is an acceptable compromise because the simulated response of the on-screen application to your action (delivered visually) is sufficient to close a perceptual gap, to fool you into believing that you are actually touching a button or icon; the visual feedback serves as proxy for the missing tactile interaction with the object. However, in some situations, and for some users, the presence of the glass barrier causes the interaction to fail completely. For blind users in particular, for whom closing a perceptual loop through vision is not an option, other means of interaction with computationally mediated environments must be developed. The primary objective of this exploratory project is to develop a full-page interactive tactile display that can render what the PIs term "malleable media" by combining touch sensing with their existing microfluidic actuators to create a full-page responsive surface. Such a display could provide an immediate tangible response in reaction to how it is being touched. By transcending the glass barrier of the screen, such a display would quite literally support a direct manipulation paradigm by making both interface elements, such as buttons and icons, and application content touchable. Thus, project outcomes are likely to provide greater accessibility to digital media by blind users. But the sighted community also stands to benefit. Both students and professionals working in STEM fields require access to increasingly high dimensional data that cannot be easily accessed via speech. An interactive tactile display such as that which is the focus of this work would provide such access, to blind computer users and their sighted counterparts alike.The intellectual merit of this exploratory project derives from combining two strands of existing research by the PIs. The first is their recent work on a microfluidic tactile display, while the second is their established record in applying principles of enactive cognition to the design of human-computer interfaces. Enactive cognition posits a tight coupling between our actions on the environment and our perception of how the environment is responding. With the new technology the PIs have the means of creating such an interactive surface, but it is not yet known what such interactions should "feel" like. What would it be like to sculpt an object by molding the surface of a tablet computer with one's hands? For blind users, this would provide a means for not just feeling tactile objects, but also for creating those objects. Such objects might be used as new ways of representing, for example, patterns present in complex data sets or of interacting with mathematical models.