From Human-Powered to Automated Video Description for Blind and Low Vision Users

针对盲人和低视力用户的从人力到自动视频描述

• 批准号: 10568469
• 负责人: Pooyan Fazli
• 金额: $ 64.99万
• 依托单位: ARIZONA STATE UNIVERSITY-TEMPE CAMPUS
• 依托单位国家: 美国
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2028-06-30
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10568469
• 关键词: Address; Artificial Intelligence; Breeding; Canis familiaris; Collaborations; Communication; Communities; Computer Vision Systems; Data Set; Decision Making; Diagnosis; Education; Emotional; Employment; Ensure; Environment; Ethics; Face; Feeling; Future; German population; Goals; Health; Human; Human Volunteers; Image; Individual; Internet; Intervention; Knowledge; Life; Machine Learning; Mission; Modernization; Natural Language Processing; Outcome; Output; Pathway interactions; Personal Satisfaction; Persons; Positioning Attribute; Public Health; Research; Safety; Self-Help Devices; System; Time; Training; United States; United States National Institutes of Health; Vision; Visual; Visual impairment; Work; assistive robot; blind; community engagement; computer human interaction; design; digital; diverse data; global health; human-in-the-loop; improved; innovation; large scale data; novel; open source; response; satisfaction; social; social exclusion; tool; virtual; vision rehabilitation; visual information; volunteer

Project Summary Approximately 12 million people in the United States have been diagnosed with a visual impairment. These individuals face unique challenges in our modern environment, where much critical information related to education, employment, entertainment, and community is presented in the form of digital videos. Inaccessible information can result in social exclusion or become life threatening if individuals require access to it in order to make decisions related to their health and safety. For example, in a personal or global health crisis, individuals may need to access the mass amounts of information conveyed via videos or dynamic infographics in order to make informed decisions. To address this need, the online platform YouDescribe allows blind and low vision (BLV) users to request amateur volunteers to create video descriptions, also referred to as audio descriptions (AD), of YouTube videos. However, the platform has been unable to keep up with the overwhelming demand, and 92.5% of videos on the YouDescribe user wish list remain undescribed. The overall objective of this proposal is to build an AI-driven system, suitable for use on a wide-scale, to automatically generate descriptions of online videos, as well as answer questions asked by BLV users about the content of videos. The rationale for this project is that AI-based tools are necessary to facilitate timely access to the deluge of new videos appearing on the Internet every day. The proposed work encompasses three specific aims: 1) develop an AI-based tool in collaboration with sighted describers that more efficiently produces video descriptions and increases the availability of accessible videos. The goal is to create an AI-driven NarrationBot that will decrease the time required for novice volunteers to produce video descriptions by 80%; 2) develop an AI-based tool in collaboration with BLV individuals that offers user-driven access to visual information in online videos. The goal is to develop an AI-driven QABot that allows users to pause a video, ask questions about content, and receive immediate answers (e.g., “What breed is the dog?”, “German shepherd”) that are accurate 80% of the time; and 3) develop and publicly release large-scale datasets to improve machine learning for video accessibility. These novel datasets will be used to increase the quality and accuracy of NarrationBot and QABot until AI-generated descriptions and answers need minimal intervention from human volunteers and can serve BLV users directly. The proposed research is innovative because it focuses on videos, whereas existing AI-driven efforts to address this problem have focused primarily on static photos or images. It is also one of only a few efforts to directly partner with BLV individuals to develop AI-driven systems that produce visual descriptions or answer visual questions. The proposed research is significant because it will result in open-source, AI-driven tools that will give BLV individuals unprecedented control over their ability to independently navigate the information-rich world of online videos, thus improving their health and wellbeing.

项目摘要 在美国，大约有1200万人被诊断出患有视力障碍。这些 在我们的现代环境中，个人面临着独特的挑战，许多关键信息与 教育、就业、娱乐和社区的信息以数字视频的形式呈现。不可访问 如果个人需要获取信息， 做出与他们的健康和安全有关的决定。例如，在个人或全球健康危机中， 可能需要访问通过视频或动态信息图表传达的大量信息， 做出明智的决定。为了满足这一需求，在线平台YouDescribe允许盲人和低视力者 (BLV)用户可以请求业余志愿者创建视频描述，也称为音频描述 (AD)YouTube视频。然而，该平台已经无法跟上压倒性的需求， YouDescribe用户愿望清单上92.5%的视频仍然没有描述。本提案的总体目标是 是建立一个人工智能驱动的系统，适用于大规模使用，自动生成在线描述， 视频，以及回答BLV用户关于视频内容的问题。这样做的理由 该项目的一个关键问题是，基于人工智能的工具对于及时访问出现在 每天上网。拟议的工作包括三个具体目标：1）开发一个基于AI的工具， 与视力良好的描述者合作，更有效地制作视频描述， 提供无障碍视频。我们的目标是创建一个AI驱动的NarrationBot， 要求新手志愿者制作80%的视频描述; 2）合作开发基于AI的工具 与BLV个人，提供用户驱动的访问在线视频中的视觉信息。我们的目标是开发 一个AI驱动的QABot，允许用户暂停视频，询问有关内容的问题， 答案（例如，“狗是什么品种的？”“德国牧羊犬”），准确的80%的时间;和3）发展 并公开发布大规模数据集，以改进机器学习，提高视频可访问性。这些新颖 数据集将用于提高NarrationBot和QABot的质量和准确性，直到AI生成 描述和回答需要人类志愿者的最小干预，并且可以直接为BLV用户服务。 拟议的研究是创新的，因为它专注于视频，而现有的人工智能驱动的努力，以解决 这个问题主要集中在静态照片或图像上。这也是为数不多的直接 与BLV个人合作开发人工智能驱动的系统，以产生视觉描述或回答视觉 问题.拟议的研究意义重大，因为它将产生开源的人工智能驱动的工具， BLV个人对他们独立导航信息丰富的世界的能力的前所未有的控制。 在线视频，从而改善他们的健康和福祉。