Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired

 CamIO（“相机输入-输出”）是一种新颖的相机系统，旨在使盲人或视力受损的人能够完全访问物理对象（包括文档、地图和3D对象，如建筑模型）。 它的工作原理是提供实时的音频触觉反馈，以响应用户指向的对象上的位置，这是可见的相机安装在工作空间上方。 当探索对象时，用户可以自由地移动它;诸如用手指“双击”的手势向系统发出信号，以提供关于手指当前指向的对象上的位置的音频反馈（或者对于具有低视力的用户，所选位置的增强图像）。 与其他使盲人或视障人士可以访问对象的方法相比，CamIO具有几个优点：（a）不需要修改或增强现有对象（例如，（B）CamIO甚至对那些不精通盲文的人也可以使用;（c）它允许用所有手指自然地探索物体（与那些使用盲文的方法相比， 依赖于使用特殊的触针）。 还要注意的是，现有的使触摸敏感平板电脑（如iPad）上的图形可访问的方法只能提供有限的触觉反馈-音频和振动提示-这与通过用手指探索物理对象获得的触觉反馈相比是严重贫乏的。 我们建议为CamIO开发必要的计算机视觉算法，以学习和识别对象，估计每个对象的姿势，以帮助确定手指在对象上指向的位置，跟踪手指，识别手势并对打印在对象表面上的文本执行OCR（光学字符识别）。 该系统将设计有特殊的用户界面功能，使盲人或视障用户能够自由探索感兴趣的对象，并自然地与之互动，以访问 他们想要的信息，这些信息将以适合他们需求的方式呈现（例如，文本到语音或膝上型计算机屏幕上的文本的增强图像）。 其他功能将允许视力助理（亲自或远程）提供对象注释信息。 最后，盲人或视障人士-CamIO系统的目标用户-将参与这项拟议研究的各个方面，以最大限度地发挥研究工作的影响。 在赠款结束时，我们计划将CamIO软件作为一个自由和开源（FOSS）项目发布，任何人都可以下载和使用，并且可以自由构建或修改以用于第三方软件。

 DESCRIPTION (provided by applicant): Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary CamIO ("Camera Input-Output") is a novel camera system designed to make physical objects (including documents, maps and 3D objects such as architectural models) fully accessible to people who are blind or visually impaired. It works by providing real-time audio-haptic feedback in response to the location on an object that the user is pointing to, which is visible to the camera mounted above the workspace. While exploring the object, the user can move it freely; a gesture such as "double-tapping" with a finger signals for the system to provide audio feedback about the location on the object currently pointed to by the finger (or an enhanced image of the selected location for users with low vision). Compared with other approaches to making objects accessible to people who are blind or visually impaired, CamIO has several advantages: (a) there is no need to modify or augment existing objects (e.g., with Braille labels or special touch-sensitive buttons), requiring only a low- cost camera and laptop computer; (b) CamIO is accessible even to those who are not fluent in Braille; and (c) it permits natural exploration of the object with all fingers (in contrast with approaches that rely on the use of a special stylus). Note also that existing approaches to making graphics on touch-sensitive tablets (such as the iPad) accessible can provide only limited haptic feedback - audio and vibration cues - which is severely impoverished compared with the haptic feedback obtained by exploring a physical object with the fingers. We propose to develop the necessary computer vision algorithms for CamIO to learn and recognize objects, estimate each object's pose to help determine where the fingers are pointing on the object, track the fingers, recognize gestures and perform OCR (optical character recognition) on text printed on the object surface. The system will be designed with special user interface features that enable blind or visually impaired users to freely explore an object of interest and interact with it naturally to access the information they want, which will be presented in a modality appropriate for their needs (e.g., text-to-speech or enhanced images of text on the laptop screen). Additional functions will allow sighted assistants (either in person or remote) to contribute object annotation information. Finally, people who are blind or visually impaired - the target users of the CamIO system - will be involved in all aspects of this proposed research, to maximize the impact of the research effort. At the conclusion of the grant we plan to release CamIO software as a free and open source (FOSS) project that anyone can download and use, and which can be freely built on or modified for use in 3rd-party software.