CAREER: Raster Multiview Algebra for Unlabeled Visual Data Exploration

各种类型的相机在我们的环境中不断增加，不断记录我们的日常行为以及与周围物体、场景和其他人的互动。然而，这种不断增长的收集视觉数据的能力，并没有与个人被记录（由他们自己或他人）测量、分析和预测这些数据的能力的可比加速相匹配。该项目开发了一种计算机视觉系统，可以执行从个人视觉数据中学习的任务，例如通过用户的可穿戴摄像头进行饮食监测。该项目将开发一种通用的3D几何表示来编码多视图图像流，该图像流可以提供辅助的空间约束，以实现半监督学习。这种表示将通过利用大量或潜在无限数量的未标记的多视图图像流来实现许多计算机视觉任务。研究者计划通过多种教育活动传播这项研究，包括：(a)通过在明尼苏达大学举办的暑期技术营、女孩机器学习日营和机器人展的一系列讲习班，向代表性不足群体的K-12学生传播这项研究；(b)每年在明尼苏达州博览会上公开展示研究成果；(c)本科生通过开发短期课程材料参与研究；(d)整合多视图几何和视觉识别（深度学习）的栅格多视图代数新课程开发；(e)透过举办电脑视觉方面的辅导及工作坊，传播研究成果。本研究探讨了一种新的多视图几何的栅格表示，它提供了3D重建与视觉识别模型训练的紧密集成。该理论将允许以两种方式广泛利用未标记的多视图图像流：(1)通过以无监督的方式通过3D重建积极探索多视图视觉数据，以及(2)通过几何方式跨视图传递概率信念进行跨视图监督。该项目将通过一种新颖的光栅束调整将这一理论推广到未校准的移动摄像机。该研究项目将为两个主要科学学科奠定计算基础：(1)行为科学：栅格理论将解决许多定制的视觉任务，以表征微观社会信号，使我们能够克服现有方法在高危儿童（如自闭症谱系障碍，通常使用主观和零星的测量方法进行评估）的行为评估的基本局限性；(2)神经科学：计划中的研究将使灵长类动物自由放养活动的计算测量成为可能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Cameras of all types continue to proliferate in our environments, continuously recording our everyday actions and interactions with surrounding objects, scenes, and other people. This growing capacity to collect visual data has not, however, been matched by a comparable acceleration in the ability of individuals being recorded (by themselves or others) to measure, analyze, and predict from this data as they desire. This project develops a computer vision system that performs tasks of learning from personal visual data, e.g., dietary monitoring via a user's wearable cameras. The project will develop a common 3D geometry representation to encode multiview image streams that can provide an auxiliary spatial constraint to enable semi-supervised learning. This representation will enable many computer vision tasks by leveraging a large or potentially infinite number of unlabeled multiview image streams. The investigator plans to disseminate this research through multiple educational activities, including: (a) K-12 students of under-represented groups through a series of workshops in Summer Technology Camp, Girls' Machine Learning Day Camp, and the Robot Show at the University Minnesota; (b) public demonstration of the research outcomes annually at the Minnesota State Fair; (c) undergraduate research involvement through short course material developments; (d) new curriculum development on the raster multiview algebra that consolidates multiview geometry and visual recognition (deep learning); and (e) dissemination of research findings through tutorial and workshop organization for broader audiences in computer vision.This research investigates a new raster representation of multiview geometry that offers a tight integration of 3D reconstruction with the training of a visual recognition model. This theory will allow extensive utilization of the unlabeled multiview image streams in two ways: (1) by actively exploring multiview visual data through 3D reconstruction in an unsupervised manner, and (2) by geometrically transferring a probabilistic belief across views for cross-view supervision. The project will generalize this theory to uncalibrated moving cameras through a novel raster bundle adjustment. This research program will lay a computational foundation for two major scientific disciplines: (1) Behavioral science: the raster theory will address many customized visual tasks to characterize microscopic social signals, enabling us to overcome the fundamental limitations of existing approaches in behavioral assessments for at-risk children (such as those with autism spectrum disorder, which have typically been assessed using a subjective and sporadic measure); and (2) Neuroscience: the planned research will enable computational measurement of free-ranging activities of primate subjects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.