CAREER: Raster Multiview Algebra for Unlabeled Visual Data Exploration

各种类型的相机继续在我们的环境中激增，不断记录我们的日常行动以及与周围物体、场景和其他人的互动。然而，这种不断增长的收集视觉数据的能力并没有与被记录的个人(由他们自己或其他人)根据他们想要的从这些数据测量、分析和预测的能力相匹配的加速。该项目开发了一种计算机视觉系统，该系统执行从个人视觉数据中学习的任务，例如通过用户的可穿戴相机进行饮食监测。该项目将开发一种通用的3D几何表示法来编码多视点图像流，该图像流可以提供辅助空间约束以实现半监督学习。这种表示将通过利用大量或潜在无限数量的未标记多视点图像流来实现许多计算机视觉任务。调查员计划通过多种教育活动传播这项研究，其中包括：(A)通过在暑期技术营、女童机器学习日营和明尼苏达大学机器人展举办的一系列讲习班，宣传这项研究成果；(B)每年在明尼苏达州博览会上公开展示研究成果；(C)通过编写短期课程材料，让本科生参与研究；(D)开发关于整合多视点几何和视觉识别的栅格多视点代数的新课程(深度学习)；以及(E)通过指导和研讨会组织向更广泛的计算机视觉受众传播研究成果。这项研究调查了一种新的多视图几何的栅格表示，该表示提供了3D重建与视觉识别模型的训练的紧密集成。这一理论将允许以两种方式广泛地利用未标记的多视点图像流：(1)通过以无监督的方式通过3D重建来主动地探索多视点视觉数据；(2)通过几何地跨视点传递概率信念来进行交叉视点监督。该项目将通过一种新的光栅束平差将这一理论推广到未校准的移动相机。这项研究计划将为两个主要科学学科奠定计算基础：(1)行为科学：栅格理论将解决许多定制的视觉任务来表征微观社会信号，使我们能够克服现有方法在高危儿童行为评估中的基本限制(例如那些患有自闭症谱系障碍的儿童，通常使用主观和零星的测量进行评估)；以及(2)神经科学：计划中的研究将能够计算测量灵长类受试者的自由活动。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Cameras of all types continue to proliferate in our environments, continuously recording our everyday actions and interactions with surrounding objects, scenes, and other people. This growing capacity to collect visual data has not, however, been matched by a comparable acceleration in the ability of individuals being recorded (by themselves or others) to measure, analyze, and predict from this data as they desire. This project develops a computer vision system that performs tasks of learning from personal visual data, e.g., dietary monitoring via a user's wearable cameras. The project will develop a common 3D geometry representation to encode multiview image streams that can provide an auxiliary spatial constraint to enable semi-supervised learning. This representation will enable many computer vision tasks by leveraging a large or potentially infinite number of unlabeled multiview image streams. The investigator plans to disseminate this research through multiple educational activities, including: (a) K-12 students of under-represented groups through a series of workshops in Summer Technology Camp, Girls' Machine Learning Day Camp, and the Robot Show at the University Minnesota; (b) public demonstration of the research outcomes annually at the Minnesota State Fair; (c) undergraduate research involvement through short course material developments; (d) new curriculum development on the raster multiview algebra that consolidates multiview geometry and visual recognition (deep learning); and (e) dissemination of research findings through tutorial and workshop organization for broader audiences in computer vision.This research investigates a new raster representation of multiview geometry that offers a tight integration of 3D reconstruction with the training of a visual recognition model. This theory will allow extensive utilization of the unlabeled multiview image streams in two ways: (1) by actively exploring multiview visual data through 3D reconstruction in an unsupervised manner, and (2) by geometrically transferring a probabilistic belief across views for cross-view supervision. The project will generalize this theory to uncalibrated moving cameras through a novel raster bundle adjustment. This research program will lay a computational foundation for two major scientific disciplines: (1) Behavioral science: the raster theory will address many customized visual tasks to characterize microscopic social signals, enabling us to overcome the fundamental limitations of existing approaches in behavioral assessments for at-risk children (such as those with autism spectrum disorder, which have typically been assessed using a subjective and sporadic measure); and (2) Neuroscience: the planned research will enable computational measurement of free-ranging activities of primate subjects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.