RII Track-2 FEC: The Visual Experience Database: A Large-Scale Point-of-View Video Database for Vision Research

RII Track-2 FEC：视觉体验数据库：用于视觉研究的大规模视点视频数据库

• 批准号: 1920896
• 负责人: Michelle Greene
• 金额: $ 397.4万
• 依托单位: Bates College
• 依托单位国家: 美国
• 项目类别: Cooperative Agreement
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1920896&HistoricalAwards=false
• 关键词: RII; Track; FEC; Visual; Experience

Current artificial intelligence (AI) systems that recognize visual content require millions of training examples to achieve good performance. However, the databases used to train such systems often take photos and videos from the internet, and thus do not represent the content that humans see on a daily basis. This introduces substantial biases into the AI systems that can have serious implications for AI-based applications such as self-driving cars. This project, a collaboration between Bates College, the University of Nevada, Reno, and North Dakota State University, Fargo will create the Visual Experience Database (VED), a database of over 240 hours of video shot from the perspective of a diverse set of observers engaged in common, everyday activities such as shopping, eating, or walking. Along with these videos, we will track each observer's head and eye position in order to understand how people look at the world, and how this changes with environment, age, and task. Our goal is to make this database open and accessible to all. Having the computer skills to use the database is key to accessibility, so we will be releasing a suite of software tools for using the database, as well as implement a summer workshop in basic computer programming skills to grow a workforce that is prepared for a variety of scientific occupations. By making the database open to the public, we will enable scientists, historians, and even artists to benefit from this rich resource. Progress in both human visual neuroscience and computer vision are limited by the availability of representative visual data. However, currently available image and movie databases are not representative of typical first-person visual experience. This project, a collaboration between Bates College, the University of Nevada, Reno, and North Dakota State University, Fargo, will create the Visual Experience Database (VED), a database of over 240 hours of first-person video complete with eye- and head-tracking. We will record from people of diverse ages (5-70 years) across three geographically distinct sites as they engage in common, everyday activities such as shopping, eating, or walking. With these data, we will be able to assess how observers sample their visual environments, and how gaze patterns change with environment, age, and task. Further, these data can be used as training data for next-generation computer vision systems. In order to develop a workforce with the skills necessary to work with big data, we will teach a Big Data Skills Summer Workshop to provide undergraduate and graduate students with the basic skills of computer programming and computational literacy to make contributions to this project and to prepare them for a variety of STEM occupations. The VED will be of broad use across several academic communities (cognitive science, neuroscience, computer vision, and possibly digital humanities and art). By creating a database that represents common, human experiences, we bypass the many biases of extant datasets, which will increase the efficacy of computer vision algorithms. By making these data fully open, we will enable advances in these fields to be accessible to all.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当前识别视觉内容的人工智能（AI）系统需要数百万个训练示例才能实现良好的性能。然而，用于训练此类系统的数据库通常从互联网上获取照片和视频，因此并不代表人类每天看到的内容。这给人工智能系统带来了很大的偏差，可能会对基于人工智能的应用（如自动驾驶汽车）产生严重影响。这个项目是贝茨学院、内华达州大学、里诺大学和北达科他州州立大学法戈大学之间的合作项目，法戈大学将创建视觉体验数据库（VED），这是一个包含240多个小时视频的数据库，这些视频是从不同观察者的角度拍摄的，这些观察者从事共同的日常活动，如购物、吃饭或散步。沿着这些视频，我们将跟踪每个观察者的头部和眼睛位置，以了解人们如何看待世界，以及这种情况如何随着环境，年龄和任务而变化。我们的目标是使这一数据库向所有人开放和访问。拥有使用数据库的计算机技能是访问的关键，因此我们将发布一套使用数据库的软件工具，并实施基本计算机编程技能的夏季研讨会，以培养为各种科学职业做好准备的劳动力。通过向公众开放数据库，我们将使科学家，历史学家，甚至艺术家从这一丰富的资源中受益。人类视觉神经科学和计算机视觉的进展受到代表性视觉数据可用性的限制。然而，目前可用的图像和电影数据库并不代表典型的第一人称视觉体验。这个项目是贝茨学院、内华达州大学、里诺和北达科他州州立大学法戈分校之间的合作，将创建视觉体验数据库（VED），这是一个包含240多个小时第一人称视频的数据库，并配有眼球和头部跟踪。我们将记录不同年龄段（5-70岁）的人在三个地理位置不同的网站，因为他们从事共同的日常活动，如购物，吃饭，或步行。有了这些数据，我们将能够评估观察者如何对他们的视觉环境进行采样，以及注视模式如何随环境、年龄和任务而变化。此外，这些数据可以用作下一代计算机视觉系统的训练数据。为了培养具有使用大数据所需技能的劳动力，我们将教授大数据技能夏季研讨会，为本科生和研究生提供计算机编程和计算素养的基本技能，为该项目做出贡献，并为各种STEM职业做好准备。VED将在多个学术团体（认知科学、神经科学、计算机视觉，可能还有数字人文和艺术）中广泛使用。通过创建一个代表共同的人类经验的数据库，我们绕过了现有数据集的许多偏见，这将提高计算机视觉算法的效率。通过完全开放这些数据，我们将使所有人都能获得这些领域的进步。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Characterizing the Performance of Deep Neural Networks for Eye-Tracking

表征眼动追踪深度神经网络的性能

• DOI: 10.1145/3450341.3458491
• 发表时间: 2021
• 期刊: Eye Tracking Research and Applications
• 作者: Biswas, Arnab;Binaee, Kamran;Capurro, Kaylie Jacleen;Lescroart, Mark D.
• 通讯作者: Lescroart, Mark D.

Pupil Tracking Under Direct Sunlight

直射阳光下的瞳孔跟踪

• DOI: 10.1145/3450341.3458490
• 发表时间: 2021
• 期刊: Eye Tracking Research and Applications
• 作者: Binaee, Kamran;Sinnott, Christian;Capurro, Kaylie Jacleen;MacNeilage, Paul;Lescroart, Mark D
• 通讯作者: Lescroart, Mark D

VEDBViz: The Visual Experience Database Visualization and Interaction Tool

VEDBViz：视觉体验数据库可视化和交互工具

• DOI: 10.1145/3450341.3458486
• 发表时间: 2021
• 期刊: Eye Tracking Research and Applications
• 作者: Ramanujam, Sanjana;Sinnott, Christian;Shankar, Bharath;Halow, Savannah Jo;Szekely, Brian;MacNeilage, Paul;Binaee, Kamran
• 通讯作者: Binaee, Kamran