CRII: CHS: Predicting When, Why, and How Multiple People Will Disagree when Answering a Visual Question

可视化问答系统的目标是使人们能够找到关于任何图像的任何问题的答案。例如，VQA系统可以让盲人解决日常的视觉挑战，比如了解一双袜子是否匹配，或者了解罐头里是什么类型的食物。VQA服务还可以促进创建更智能的环境，例如在任何给定时间监控工厂装配线上有多少缺陷产品。现有VQA系统的一个局限性是它们没有考虑到这样一个事实，即一个视觉问题可能会引起不同人的不同答案。如果VQA系统使用户能够预测和解决可能出现的任何答案分歧，则可以节省时间并减少用户的挫败感。盲人和有视力的人可以更快、更准确地了解人类对视觉世界的多样性观点。VQA服务还可以教人们如何提出视觉问题，以获得所需答案的多样性。该项目将创建人工智能(AI)模型，可以解释人群智能固有的可能答案的多样性。具体地说，人工智能模型将被设计成预测人类回答分歧的时间、原因和方式，这反过来将使人机合作伙伴关系的新设计成为可能。这是具有挑战性的，因为它需要设计框架，同时模拟和综合对图像和语言的不同和潜在冲突的看法，以解决许多可能导致分歧的原因。为了确保人工智能模型在广泛的应用中得到推广，现有的由盲人和有视力的人提出的100多万个视觉问题的语料库将被用来创建带注释的数据集，这些数据集表明何时、为什么以及出现多少答案不一致。然后，将开发方法，直接根据视觉问题自动预测一个群体将产生多少答案多样性，以及当答案多样化时为什么会出现分歧。最后，将设计一个系统，引导视障用户更快地提出视觉问题，以便他们能够收到单一、明确的人群反应(例如，引导人们用手机摄像头更好地框住感兴趣的视觉内容)。将对盲人用户进行用户研究，以经验性地测试新系统的有效性，重点是在现实世界、实时情况下揭示以人为本的问题。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of a visual question answering (VQA) system is to empower people to find the answer to any question about any image. For example, a VQA system could enable blind people to address daily visual challenges such as learning whether a pair of socks match or learning what type of food is in a can. VQA services could also facilitate the creation of smarter environments, say to monitor how many defective products are on a factory assembly line at any given time. A limitation of existing VQA systems is that they do not account for the fact that a visual question may elicit different answers from different people. VQA systems could save time and reduce user frustration if they empowered users to anticipate and resolve any answer disagreements that may arise. Blind and sighted people could more rapidly and accurately learn about the diversity of human perspectives on the visual world. VQA services also could teach people how to ask visual questions that elicit the desired answer diversity.This project will create artificial intelligence (AI) models that can account for the possible diversity of answers inherent in crowd intelligence. Specifically, AI models will be designed to predict when, why, and how human answer disagreement occurs, which in turn will enable new designs for human-computer partnerships. This is challenging because it necessitates designing frameworks that simultaneously model and synthesize different and potentially conflicting perceptions of images and language for the many possible causes of disagreement. To ensure that the AI models generalize across a broad range of applications, an existing corpus of over one million visual questions asked by blind and sighted people will be used to create annotated datasets that indicate when, why, and how much answer disagreement arises. Methods will then be developed for automatically predicting directly from a visual question how much answer diversity will arise from a crowd, and why disagreement arises when it does. Finally, a system will be designed for guiding visually-impaired users to more quickly formulate visual questions so they can receive a single, unambiguous crowd response (e.g., guide the person to better frame the visual content of interest with a mobile phone camera). User studies with blind users will be conducted to empirically test the efficacy of the new system, with a focus on uncovering human-based issues in real-world, real-time situations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.