EAGER: Collaborative Research: Malleable Media to Support Interaction through Bi-Directional Touch Displays

与触摸屏的交互之所以引人注目，部分原因是它复制了与我们环境中物体的交互;你可以使用触摸屏来视觉上理解、接触、触摸和操纵物体。 当然，这只是部分正确，因为在最后一刻，你的手接触玻璃，其余的操作是模拟的。 对于许多应用程序来说，玻璃屏障是一个可以接受的折衷方案，因为屏幕上应用程序对您的操作的模拟响应（视觉传递）足以弥补感知差距，让您相信您实际上正在触摸按钮或图标;视觉反馈可以作为与对象的触觉交互的代理。 然而，在某些情况下，对于某些用户，玻璃屏障的存在会导致交互完全失败。 特别是对于盲人用户来说，通过视觉闭合感知回路不是一种选择，必须开发与计算介导的环境交互的其他手段。 这个探索性项目的主要目标是开发一个全页交互式触觉显示器，通过将触摸传感与现有的微流体致动器相结合，创建一个全页响应表面，可以呈现PI所称的“可延展媒体”。 这样的显示器可以提供对它如何被触摸的反应的即时有形响应。 通过超越屏幕的玻璃屏障，这样的显示器将通过使界面元素（例如按钮和图标）和应用内容都可触摸来完全支持直接操纵范例。 因此，项目成果很可能为盲人用户提供更大的数字媒体无障碍环境。 但视力良好的社区也会受益。 在STEM领域工作的学生和专业人士都需要访问越来越高维的数据，这些数据无法通过语音轻松访问。这项工作的重点是一种交互式触觉显示器，它将为盲人和视力正常的计算机用户提供这种访问。这项探索性项目的智力价值来自于结合了PI现有的两种研究。 第一个是他们最近在微流体触觉显示器上的工作，第二个是他们在将生成认知原理应用于人机界面设计方面的既定记录。 主动认知假设我们对环境的行为与我们对环境如何反应的感知之间存在紧密耦合。 有了新技术，PI有办法创建这样一个交互式表面，但还不知道这种交互应该是什么样的“感觉”。 用手塑造平板电脑的表面来雕刻一个物体会是什么样子？ 对于盲人用户来说，这将提供一种不仅能感知触觉对象，还能创建这些对象的方法。 这样的对象可能被用作新的方式来表示，例如，复杂数据集中存在的模式或与数学模型交互。

Interaction with touchscreens is compelling because, in part, it replicates interaction with objects in our environment; you can use a touchscreen to visually apprehend, reach toward, touch and manipulate an object. Of course, this is only partly true because, at the last moment, your hand contacts glass and the rest of the manipulation is simulated. For many applications, the glass barrier is an acceptable compromise because the simulated response of the on-screen application to your action (delivered visually) is sufficient to close a perceptual gap, to fool you into believing that you are actually touching a button or icon; the visual feedback serves as proxy for the missing tactile interaction with the object. However, in some situations, and for some users, the presence of the glass barrier causes the interaction to fail completely. For blind users in particular, for whom closing a perceptual loop through vision is not an option, other means of interaction with computationally mediated environments must be developed. The primary objective of this exploratory project is to develop a full-page interactive tactile display that can render what the PIs term "malleable media" by combining touch sensing with their existing microfluidic actuators to create a full-page responsive surface. Such a display could provide an immediate tangible response in reaction to how it is being touched. By transcending the glass barrier of the screen, such a display would quite literally support a direct manipulation paradigm by making both interface elements, such as buttons and icons, and application content touchable. Thus, project outcomes are likely to provide greater accessibility to digital media by blind users. But the sighted community also stands to benefit. Both students and professionals working in STEM fields require access to increasingly high dimensional data that cannot be easily accessed via speech. An interactive tactile display such as that which is the focus of this work would provide such access, to blind computer users and their sighted counterparts alike.The intellectual merit of this exploratory project derives from combining two strands of existing research by the PIs. The first is their recent work on a microfluidic tactile display, while the second is their established record in applying principles of enactive cognition to the design of human-computer interfaces. Enactive cognition posits a tight coupling between our actions on the environment and our perception of how the environment is responding. With the new technology the PIs have the means of creating such an interactive surface, but it is not yet known what such interactions should "feel" like. What would it be like to sculpt an object by molding the surface of a tablet computer with one's hands? For blind users, this would provide a means for not just feeling tactile objects, but also for creating those objects. Such objects might be used as new ways of representing, for example, patterns present in complex data sets or of interacting with mathematical models.