RI: Small: Exploring Rationale behind Visual Understanding: Combining Attention and Reasoning

RI：小：探索视觉理解背后的基本原理：注意力和推理的结合

• 批准号: 1908711
• 负责人: Qi Zhao
• 金额: $ 28.5万
• 依托单位: University of Minnesota-Twin Cities
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908711&HistoricalAwards=false
• 关键词: RI; Small; Exploring; Rationale; behind

Recent progress in deep learning has resulted in models that show significant performance gains in computer vision tasks. This project aims to bridge the current gap between the increasing performance in intelligent systems and the lack of understanding in the complex task-solving process. With the overarching goal of understanding and modeling the process, this project studies two intertwined mechanisms heavily involved in task-solving -- attention and reasoning -- and develops a sound framework to integrate the two. It will serve as a critical step forward to untangling the process of solving a visual task and alleviating the black-box problem in machine learning. The research will build attention and reasoning capabilities into machines, thus empowering applications in a broad spectrum of artificial intelligence tasks including medical diagnosis and treatment, robotics, and education. The principal investigator will organize workshops and seminars, and make project results publicly available. The project also aims at integrated research and education with a focus on increased diversity, through K-12 outreach activities, student mentoring, and curriculum development.This project focuses on both dataset and model development, as well as enabling new methods for network visualization, interpretation, and diagnosis. More specifically, the project develops: (1) a new dataset with human eye movements and textual explanations, to understand critical factors that contribute to task performance; (2) a framework where models devised in the framework make a first step to demonstrate the process of task-solving by showing attention and reasoning capabilities; and (3) a novel layer-wise network diagnosis method considering both performance and interpretability of each network layer. Addressing these questions will not only boost model performance but open the black-box of the decision-making process of a visual task as well as the structure of the deep neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习的最新进展已经导致模型在计算机视觉任务中显示出显着的性能提升。该项目旨在弥合目前智能系统性能不断提高与对复杂任务解决过程缺乏理解之间的差距。随着理解和建模的过程的总体目标，这个项目研究了两个相互交织的机制，大量参与任务解决-注意力和推理-并开发了一个健全的框架来整合两者。它将成为解决视觉任务和缓解机器学习中黑盒问题的关键一步。该研究将在机器中建立注意力和推理能力，从而为广泛的人工智能任务提供应用，包括医疗诊断和治疗，机器人和教育。首席研究员将组织讲习班和研讨会，并公布项目成果。该项目还旨在通过K-12外展活动、学生辅导和课程开发，实现以增加多样性为重点的综合研究和教育。该项目侧重于数据集和模型开发，以及实现网络可视化、解释和诊断的新方法。更具体地说，该项目开发了：（1）一个新的数据集，包含人眼运动和文本解释，以了解有助于任务绩效的关键因素;（2）一个框架，在该框架中设计的模型通过显示注意力和推理能力来展示任务解决过程的第一步;（3）综合考虑网络各层性能和可解释性的分层网络诊断方法。解决这些问题不仅可以提高模型性能，还可以打开视觉任务决策过程的黑匣子以及深度神经网络的结构。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Query and Attention Augmentation for Knowledge-Based Explainable Reasoning

• DOI: 10.1109/cvpr52688.2022.01513
• 发表时间: 2022-06
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Yifeng Zhang;Ming Jiang;Qi Zhao

Leveraging Human Attention in Novel Object Captioning

• DOI: 10.24963/ijcai.2021/86
• 发表时间: 2021-08
• 作者: Xianyu Chen;Ming Jiang;Qi Zhao

Self-Distillation for Few-Shot Image Captioning

• DOI: 10.1109/wacv48630.2021.00059
• 发表时间: 2021-01
• 期刊: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Xianyu Chen;Ming Jiang;Qi Zhao

REX: Reasoning-aware and Grounded Explanation

• DOI: 10.1109/cvpr52688.2022.01514
• 发表时间: 2022-03
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Shi Chen;Qi Zhao

Learning to Minimize the Remainder in Supervised Learning

• DOI: 10.1109/tmm.2022.3158066
• 发表时间: 2022-01
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Yan Luo;Yongkang Wong;Mohan S. Kankanhalli;Qi Zhao-