CRII: CHS: Predicting When, Why, and How Multiple People Will Disagree when Answering a Visual Question

CRII：CHS：预测多人在回答视觉问题时何时、为何以及如何产生分歧

• 批准号: 1755593
• 负责人: Danna Gurari
• 金额: $ 17.49万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-05-01 至 2021-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1755593&HistoricalAwards=false
• 关键词: CRII; CHS; Predicting; When; Why

The goal of a visual question answering (VQA) system is to empower people to find the answer to any question about any image. For example, a VQA system could enable blind people to address daily visual challenges such as learning whether a pair of socks match or learning what type of food is in a can. VQA services could also facilitate the creation of smarter environments, say to monitor how many defective products are on a factory assembly line at any given time. A limitation of existing VQA systems is that they do not account for the fact that a visual question may elicit different answers from different people. VQA systems could save time and reduce user frustration if they empowered users to anticipate and resolve any answer disagreements that may arise. Blind and sighted people could more rapidly and accurately learn about the diversity of human perspectives on the visual world. VQA services also could teach people how to ask visual questions that elicit the desired answer diversity.This project will create artificial intelligence (AI) models that can account for the possible diversity of answers inherent in crowd intelligence. Specifically, AI models will be designed to predict when, why, and how human answer disagreement occurs, which in turn will enable new designs for human-computer partnerships. This is challenging because it necessitates designing frameworks that simultaneously model and synthesize different and potentially conflicting perceptions of images and language for the many possible causes of disagreement. To ensure that the AI models generalize across a broad range of applications, an existing corpus of over one million visual questions asked by blind and sighted people will be used to create annotated datasets that indicate when, why, and how much answer disagreement arises. Methods will then be developed for automatically predicting directly from a visual question how much answer diversity will arise from a crowd, and why disagreement arises when it does. Finally, a system will be designed for guiding visually-impaired users to more quickly formulate visual questions so they can receive a single, unambiguous crowd response (e.g., guide the person to better frame the visual content of interest with a mobile phone camera). User studies with blind users will be conducted to empirically test the efficacy of the new system, with a focus on uncovering human-based issues in real-world, real-time situations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

视觉问答（VQA）系统的目标是使人们能够找到关于任何图像的任何问题的答案。 例如，VQA系统可以使盲人解决日常视觉挑战，例如学习一双袜子是否匹配或学习罐头中的食物类型。 VQA服务还可以促进创建更智能的环境，例如监控工厂装配线上在任何给定时间有多少缺陷产品。 现有的VQA系统的局限性在于它们没有考虑到视觉问题可能从不同的人得到不同的答案的事实。 VQA系统可以节省时间，减少用户的挫折感，如果他们授权用户预测和解决任何可能出现的答案分歧。 盲人和有视力的人可以更快、更准确地了解人类对视觉世界的不同看法。 VQA服务还可以教人们如何提出视觉问题，从而获得所需的答案多样性。该项目将创建人工智能（AI）模型，可以解释群体智能中固有的答案多样性。 具体来说，人工智能模型将被设计用于预测人类回答不一致的时间、原因和方式，这反过来将为人机合作关系提供新的设计。 这是具有挑战性的，因为它需要设计的框架，同时建模和合成不同的和潜在的冲突的图像和语言的许多可能的原因不一致的看法。 为了确保人工智能模型在广泛的应用中推广，盲人和视力正常的人提出的超过100万个视觉问题的现有语料库将用于创建注释数据集，以指示何时，为什么以及有多少答案出现分歧。 然后，将开发方法来自动预测直接从一个视觉问题有多少答案的多样性将出现在人群中，以及为什么会出现分歧时，它。 最后，一个系统将被设计用于指导视力受损的用户更快地制定视觉问题，这样他们就可以收到一个单一的，明确的人群响应（例如，引导人用移动的电话相机更好地框定感兴趣的视觉内容）。 将对盲人用户进行用户研究，以实证方式测试新系统的有效性，重点是揭示现实世界中实时情况下的人为问题。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

VizWiz Grand Challenge: Answering Visual Questions from Blind People

• DOI: 10.1109/cvpr.2018.00380
• 发表时间: 2018-02
• 期刊: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
• 作者: D. Gurari;Qing Li;Abigale Stangl;Anhong Guo;Chi Lin;K. Grauman;Jiebo Luo;Jeffrey P. Bigham

Visual Question Answer Diversity

• DOI: 10.1609/hcomp.v6i1.13341
• 发表时间: 2018-06
• 作者: Chun-Ju Yang;K. Grauman;D. Gurari

Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image

• DOI: 10.1109/wacv.2019.00166
• 发表时间: 2018-03
• 期刊: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Yinan Zhao;Brian L. Price;Scott D. Cohen;D. Gurari

BrowseWithMe: An Online Clothes Shopping Assistant for People with Visual Impairments

BrowseWithMe：为视障人士提供的在线服装购物助手

• DOI: 10.1145/3234695.3236337
• 发表时间: 2018
• 期刊: ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Stangl, Abigale J.;Kothari, Esha;Jain, Suyog D.;Yeh, Tom;Grauman, Kristen;Gurari, Danna
• 通讯作者: Gurari, Danna

Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch

• DOI: 10.1007/s11263-019-01172-6
• 发表时间: 2019-09-01
• 期刊: INTERNATIONAL JOURNAL OF COMPUTER VISION
• 影响因子: 19.5
• 作者: Gurari, Danna;Zhao, Yinan;Grauman, Kristen
• 通讯作者: Grauman, Kristen