Glove-based Tactile Streaming of Braille Characters and Digital Images for the Visually Impaired

为视障人士提供基于手套的盲文字符和数字图像触觉流传输

• 批准号: 10601900
• 负责人: Michel Albert Audette
• 金额: $ 25.72万
• 依托单位: HAPTX INC.
• 依托单位国家: 美国
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-30 至 2024-09-29
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10601900
• 关键词: 12 year old; 3-Dimensional; Address; Adult; Age; Algorithmic Software; Blindness; Braille Display; Calibration; Characteristics; Child; Collection; Communities; Computer software; Consultations; Coupling; Data; Development; Encapsulated; Endowment; Engineering; Eye diseases; Fingers; Foundations; Fracture; Genetic Transcription; Goals; Growth; Hand; Health; Image; Individual; Label; Length; Manuals; Maps; Mental Depression; Modality; Monitor; Multimedia; Network-based; Optometrist; Palpable; Paper; Patients; Perception; Phase; Population; Prevalence; Process; Quality of life; Readability; Reader; Reading; Research; Research Personnel; Resolution; Risk; Sculpture; Semantics; Stream; Students; Surface; System; Tablets; Tactile; Technology; Text; Thumb structure; Time; Validation; Vision; Visual Acuity; Visual impairment; age related; aging population; blind; braille; cohort; computerized; deep learning; deep neural network; demographics; density; design; digital; digital imaging; disability; fall risk; haptic feedback; haptic technology; haptics; indexing; lexical; multimodality; neural network; object recognition; open source; pressure; spatiotemporal; validation studies; virtual

This project will achieve a computerized system that supplies digital content, both lexical and graphical, suitable for children, students and working adults of the Visually Impaired (VId) community, to increase their autonomy and potentiate their Quality of Life and societal contribution. While there is notable academic research, such as a high-density pin display tablet, there is no commercial, broadly functional, full-page Braille reader. A tablet reader is potentially limiting in comparison with bimanual haptic gloves, limiting any content complementary to Braille text. In contrast, haptic gloves support both Braille lexical content and palpable digital images. The proposed system will encapsulate a full-length Braille reader, not limited to a page at a time but supplying a limitless stream of characters readable by four fingers in each hand as dictated by the content, while also supporting haptic actuation of embedded pictorial content. This glove-based approach will unlock, for an underserved VId clientele, a vast collection of multimedia content currently the exclusive domain of the seeing population, as Braille lexical content and palpable bas-relief of graphical content. Phase I will center on a haptic system for Braille display of full-length text, sensed by the fingers of both hands and based on a redesigned commercial bimanual glove-based haptic system. A software algorithm will stream and stagger the Braille content to each finger to replicate the bimanual reading process of a VId reader. The system will also feature a tactile bas-relief version of digital images embedded in the text. Phase II will develop Haptic VR, demonstrating the utility of overlaying Braille characters on a haptic curved surface and paving the way for vitally important haptic sensing of labeled 2D/3D graphics by VId users. A long-term goal is to apply this VR foundation to geolocated Haptic AR, enabling haptic overlay of Braille labels on everyday surfaces, from appliances to groceries, by suitable integration of haptic rendering with 3D range-sensing and DNN-based object recognition. Specific Aims – The Phase I Aims are as follows. Aim 1. High-resolution haptic gloves for Braille streaming- Development of bimanual high-resolution haptic gloves, featuring dense actuation at eight fingertips, by Haptx. The thumbs will also get haptic feedback. Aim 2. Haptic rendering of lexical and graphical content 2A. Lexical reader - A software transcription will support spatiotemporal streaming of lexical data that moves the haptic footprint of Braille characters serially from the first to the eighth finger. 2B. Graphical reader – The conversion of a digital image to a haptic bas-relief will exploit fast DNN-based image semantic segmentation, where each label will be mapped to an actuator pressure. 2C. Finger-based calibration- Both modalities will be supported by a virtual page defined on a tabletop, whose coordinates are found by a simple calibration based on three points identified by the index fingers.

该项目将实现一个计算机化的系统，提供数字内容，包括词汇和图形， 适合视障(VID)社区的儿童、学生和在职成年人使用，以增加他们的 自主性，提高他们的生活质量和社会贡献。虽然有值得注意的学术研究 研究，如高密度的针式显示平板电脑，目前还没有商用、功能广泛、整页的 盲文阅读器。与双手触觉手套相比，平板电脑阅读器具有潜在的局限性，可以限制任何 补充盲文文本的内容。相比之下，触觉手套支持盲文词汇内容和 触手可及的数字图像。 拟议的系统将封装一个全长盲文阅读器，不限于一次一页，而是 提供由内容指示的每只手中的四个手指可读的无限字符流， 同时还支持对嵌入的图片内容的触觉驱动。这种基于手套的方法将解锁， 对于服务不足的VID客户来说，大量的多媒体内容目前是 看得见的人群，作为盲文词汇内容和可触摸的图形内容的基本浮雕。 第一阶段将以触觉系统为中心，用于盲文显示全长文本，由两个人的手指感知 并基于重新设计的商用双手手套触觉系统。一种软件算法 我会将盲文内容串流和错开到每个手指上，以复制VID的双手阅读过程 读者。该系统还将在文本中嵌入数字图像的触觉半浮雕版本。 第二阶段将开发触觉虚拟现实，演示在触觉上覆盖盲文字符的用途 曲面，并为VID标记的2D/3D图形的至关重要的触觉传感铺平了道路 用户。长期目标是将这一VR基础应用于地理定位的Haptic AR，使触觉覆盖 通过适当集成触觉渲染，在日常表面(从家用电器到食品杂货)上贴上盲文标签 具有3D距离传感和基于DNN的目标识别。具体目标--第一阶段目标如下。 目的1.用于盲文流传输的高分辨率触觉手套--双手高分辨率触觉的开发 由Haptx制造的手套，具有八个指尖的密集致动功能。拇指也会得到触觉反馈。 目标2.词汇和图形内容的触觉呈现 2a.词汇阅读器-一款软件转录将支持词汇数据的时空流， 将盲文角色的触觉足迹从第一个手指顺序移动到第八个手指。 2B。图形阅读器-将数字图像转换为触觉基本浮雕将利用基于DNN的快速 图像语义分割，其中每个标签将被映射到执行器压力。 2C。基于手指的校准-两种模式都将由桌面上定义的虚拟页面支持， 其坐标是通过基于食指识别的三个点的简单校准来找到的。