CRII: CHS: Predicting When, Why, and How Multiple People Will Disagree when Answering a Visual Question

视觉问答 (VQA) 系统的目标是让人们能够找到有关任何图像的任何问题的答案。 例如，VQA 系统可以帮助盲人解决日常视觉挑战，例如了解一双袜子是否匹配或了解罐头中的食物类型。 VQA 服务还可以促进创建更智能的环境，例如监控工厂装配线上在任何给定时间有多少有缺陷的产品。 现有 VQA 系统的局限性在于，它们没有考虑到视觉问题可能会从不同的人那里引出不同的答案。 如果 VQA 系统使用户能够预测并解决可能出现的任何答案分歧，则可以节省时间并减少用户的挫败感。 盲人和视力正常的人可以更快、更准确地了解人类对视觉世界的看法的多样性。 VQA 服务还可以教人们如何提出视觉问题，从而引出所需的答案多样性。该项目将创建人工智能 (AI) 模型，该模型可以解释群体智能中固有的可能答案的多样性。 具体来说，人工智能模型将被设计为预测人类答案分歧何时、为何以及如何发生，这反过来又将为人机合作伙伴关系带来新的设计。 这是具有挑战性的，因为它需要设计一个框架，同时建模和综合不同的、潜在冲突的图像和语言感知，以解决许多可能的分歧原因。 为了确保人工智能模型能够在广泛的应用中推广，将使用由盲人和视力正常的人提出的超过一百万个视觉问题的现有语料库来创建带注释的数据集，以表明出现答案分歧的时间、原因和程度。 然后将开发方法来直接从视觉问题自动预测人群中会产生多少答案多样性，以及为什么会出现分歧。 最后，将设计一个系统来引导视障用户更快地提出视觉问题，以便他们能够收到单一、明确的人群响应（例如，引导人们更好地使用手机摄像头构建感兴趣的视觉内容）。 将针对盲人用户进行用户研究，以实证测试新系统的有效性，重点是在现实世界、实时情况下揭示人为问题。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of a visual question answering (VQA) system is to empower people to find the answer to any question about any image. For example, a VQA system could enable blind people to address daily visual challenges such as learning whether a pair of socks match or learning what type of food is in a can. VQA services could also facilitate the creation of smarter environments, say to monitor how many defective products are on a factory assembly line at any given time. A limitation of existing VQA systems is that they do not account for the fact that a visual question may elicit different answers from different people. VQA systems could save time and reduce user frustration if they empowered users to anticipate and resolve any answer disagreements that may arise. Blind and sighted people could more rapidly and accurately learn about the diversity of human perspectives on the visual world. VQA services also could teach people how to ask visual questions that elicit the desired answer diversity.This project will create artificial intelligence (AI) models that can account for the possible diversity of answers inherent in crowd intelligence. Specifically, AI models will be designed to predict when, why, and how human answer disagreement occurs, which in turn will enable new designs for human-computer partnerships. This is challenging because it necessitates designing frameworks that simultaneously model and synthesize different and potentially conflicting perceptions of images and language for the many possible causes of disagreement. To ensure that the AI models generalize across a broad range of applications, an existing corpus of over one million visual questions asked by blind and sighted people will be used to create annotated datasets that indicate when, why, and how much answer disagreement arises. Methods will then be developed for automatically predicting directly from a visual question how much answer diversity will arise from a crowd, and why disagreement arises when it does. Finally, a system will be designed for guiding visually-impaired users to more quickly formulate visual questions so they can receive a single, unambiguous crowd response (e.g., guide the person to better frame the visual content of interest with a mobile phone camera). User studies with blind users will be conducted to empirically test the efficacy of the new system, with a focus on uncovering human-based issues in real-world, real-time situations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.