Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired

为视障人士提供与物理对象的音频触觉交互

• 批准号: 9238777
• 负责人: JAMES M COUGHLAN
• 金额: $ 41.66万
• 依托单位: SMITH-KETTLEWELL EYE RESEARCH INSTITUTE
• 依托单位国家: 美国
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-03-10 至 2020-02-29
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9238777
• 关键词: Access to Information; Adoption; Algorithms; American; Architecture; Blindness; Computer Vision Systems; Computer software; Computers; Cues; Development; Economics; Employment; Ensure; Evaluation; Feedback; Fingers; Focus Groups; Gestures; Goals; Grant; Home environment; Image Enhancement; Label; Lasers; Learning; Location; Maps; Modality; Modeling; Output; Performance; Persons; Printing; Procedures; Process; Research; Rotation; Running; Schools; Science, Technology, Engineering and Mathematics; Self-Help Devices; Signal Transduction; Speech; Surface; System; Tablets; Tactile; Technology; Testing; Text; Time; Touch sensation; Training; Translations; Vision; Visual; Visual impairment; Work; Workplace; base; blind; braille; cost; design; experience; experimental study; haptics; heuristics; interest; laptop; novel; open source; optical character recognition; public health relevance; response; three-dimensional modeling; vibration

 DESCRIPTION (provided by applicant): Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary CamIO ("Camera Input-Output") is a novel camera system designed to make physical objects (including documents, maps and 3D objects such as architectural models) fully accessible to people who are blind or visually impaired. It works by providing real-time audio-haptic feedback in response to the location on an object that the user is pointing to, which is visible to the camera mounted above the workspace. While exploring the object, the user can move it freely; a gesture such as "double-tapping" with a finger signals for the system to provide audio feedback about the location on the object currently pointed to by the finger (or an enhanced image of the selected location for users with low vision). Compared with other approaches to making objects accessible to people who are blind or visually impaired, CamIO has several advantages: (a) there is no need to modify or augment existing objects (e.g., with Braille labels or special touch-sensitive buttons), requiring only a low- cost camera and laptop computer; (b) CamIO is accessible even to those who are not fluent in Braille; and (c) it permits natural exploration of the object with all fingers (in contrast with approaches that rely on the use of a special stylus). Note also that existing approaches to making graphics on touch-sensitive tablets (such as the iPad) accessible can provide only limited haptic feedback - audio and vibration cues - which is severely impoverished compared with the haptic feedback obtained by exploring a physical object with the fingers. We propose to develop the necessary computer vision algorithms for CamIO to learn and recognize objects, estimate each object's pose to help determine where the fingers are pointing on the object, track the fingers, recognize gestures and perform OCR (optical character recognition) on text printed on the object surface. The system will be designed with special user interface features that enable blind or visually impaired users to freely explore an object of interest and interact with it naturally to access the information they want, which will be presented in a modality appropriate for their needs (e.g., text-to-speech or enhanced images of text on the laptop screen). Additional functions will allow sighted assistants (either in person or remote) to contribute object annotation information. Finally, people who are blind or visually impaired - the target users of the CamIO system - will be involved in all aspects of this proposed research, to maximize the impact of the research effort. At the conclusion of the grant we plan to release CamIO software as a free and open source (FOSS) project that anyone can download and use, and which can be freely built on or modified for use in 3rd-party software.

 描述（由申请人提供）：为视障人士实现与物理对象的音频-触觉交互 摘要 CamIO（“相机输入输出”）是一种新颖的相机系统，旨在使盲人或视障人士能够完全访问物理对象（包括文档、地图和 3D 对象，例如建筑模型）。 它的工作原理是提供实时音频触觉反馈，以响应用户指向的对象的位置，该位置对于安装在工作空间上方的摄像头来说是可见的。 在探索物体时，用户可以自由移动它；诸如用手指“双击”之类的手势向系统发出信号，以提供有关手指当前指向的对象上的位置的音频反馈（或者为弱视用户提供所选位置的增强图像）。 与其他使盲人或视障人士能够接触到物体的方法相比，CamIO 有几个优点：(a) 无需修改或增强现有物体（例如，使用盲文标签或特殊的触摸感应按钮），只需要低成本的相机和笔记本电脑； (b) 即使盲文不流利的人也可以使用 CamIO； (c) 它允许用所有手指自然地探索物体（与 依赖于使用特殊的手写笔）。 还要注意的是，在触敏平板电脑（例如 iPad）上制作图形的现有方法只能提供有限的触觉反馈（音频和振动提示），与用手指探索物理对象获得的触觉反馈相比，这是严重贫乏的。 我们建议为 CamIO 开发必要的计算机视觉算法，以学习和识别物体、估计每个物体的姿势以帮助确定手指指向物体的位置、跟踪手指、识别手势并对物体表面打印的文本执行 OCR（光学字符识别）。 该系统将设计有特殊的用户界面功能，使盲人或视障用户能够自由探索感兴趣的对象并与其自然地交互以访问 他们想要的信息，将以适合他们需求的方式呈现（例如，文本转语音或笔记本电脑屏幕上的文本增强图像）。 附加功能将允许视力正常的助手（无论是亲自还是远程）提供对象注释信息。 最后，盲人或视障人士（CamIO 系统的目标用户）将参与这项拟议研究的各个方面，以最大限度地发挥研究工作的影响。 拨款结束后，我们计划将 CamIO 软件作为免费开源 (FOSS) 项目发布，任何人都可以下载和使用，并且可以自由构建或修改以在第三方软件中使用。