场景实体有向图引导下基于听觉感知的视觉替代关键技术研究

失明是影响人类生活质量最严重的残疾之一，如何防盲、治盲、助盲已是全世界关注的公共卫生问题。随着人工智能技术的发展，利用盲人听觉来替代视觉的智能助盲系统已成为当前的研究热点。然而，目前的智能助盲系统大多是将基于机器视觉的目标检测和自然语言处理进行串联堆砌，反馈的信息局限于图像中的视觉元素，缺乏一定深度的推理能力。基于计算机视觉与自然语言处理领域交叉的视觉语义描述和视觉问答为解决这一问题提供了切实可行的技术途径，但实际应用中的视觉语义多样性描述和视觉问答可控性表达仍是当前亟需攻克的问题。为此，本项目拟开展场景实体有向图引导下基于听觉感知的视觉替代关键技术研究，构建视觉语义描述和视觉问答的知识表达架构与融合新方法；并通过心理物理学实验研究，探究盲人高效执行任务时的听觉感知特征规律，进而指导模型和听觉反馈策略的优化，为未来更具“智能”的视觉替代系统设计与实现提供关键的技术理论基础和科学的实验依据。

Blindness is one of the most serious disabilities that affect the quality of human life. How to avoid, treat, and assist blindness has become a public health issue in the current world. With the rapid development of artificial intelligence technology, intelligent blindness assistance systems that use blind people’s hearing for vision substitution have become a current research focus. However, current intelligent blindness assistance systems are relied on the technologies of machine-vision-based object detection and natural language processing, which are stacked in series. Feedback information thus is limited to the visual elements in the image, and it lacks a certain depth of reasoning ability. The visual semantic description and visual question answering technologies based on the intersection of computer vision and natural language processing provide practical technical approaches to deal with this problem. However, the description of visual semantic diversity and the controllable expression of visual question answering in practical applications are still the current highland to be occupied. In response to the above, the present project intends to carry out research on key technologies of auditory-perception-based visual substitution guided by scene entity directed graph, constructing new knowledge expression framework and fusion methods for visual semantic description and visual question answering. Furthermore, we will carry out psychophysical experimental research to explore the characteristics of auditory perception when blind people perform visual tasks efficiently, and then guide the adjustment and optimization of model algorithms and auditory feedback strategies. These will provide key technical theoretical supports and scientific experimental evidence for the future design and development of a ‘smarter’ noninvasive visual substitution system.