CAREER: Raster Multiview Algebra for Unlabeled Visual Data Exploration

职业：用于无标签视觉数据探索的栅格多视图代数

• 批准号: 1846031
• 负责人: Hyun Soo Park
• 金额: $ 50.7万
• 依托单位: University of Minnesota-Twin Cities
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1846031&HistoricalAwards=false
• 关键词: CAREER; Raster; Multiview; Algebra; Unlabeled

Cameras of all types continue to proliferate in our environments, continuously recording our everyday actions and interactions with surrounding objects, scenes, and other people. This growing capacity to collect visual data has not, however, been matched by a comparable acceleration in the ability of individuals being recorded (by themselves or others) to measure, analyze, and predict from this data as they desire. This project develops a computer vision system that performs tasks of learning from personal visual data, e.g., dietary monitoring via a user's wearable cameras. The project will develop a common 3D geometry representation to encode multiview image streams that can provide an auxiliary spatial constraint to enable semi-supervised learning. This representation will enable many computer vision tasks by leveraging a large or potentially infinite number of unlabeled multiview image streams. The investigator plans to disseminate this research through multiple educational activities, including: (a) K-12 students of under-represented groups through a series of workshops in Summer Technology Camp, Girls' Machine Learning Day Camp, and the Robot Show at the University Minnesota; (b) public demonstration of the research outcomes annually at the Minnesota State Fair; (c) undergraduate research involvement through short course material developments; (d) new curriculum development on the raster multiview algebra that consolidates multiview geometry and visual recognition (deep learning); and (e) dissemination of research findings through tutorial and workshop organization for broader audiences in computer vision.This research investigates a new raster representation of multiview geometry that offers a tight integration of 3D reconstruction with the training of a visual recognition model. This theory will allow extensive utilization of the unlabeled multiview image streams in two ways: (1) by actively exploring multiview visual data through 3D reconstruction in an unsupervised manner, and (2) by geometrically transferring a probabilistic belief across views for cross-view supervision. The project will generalize this theory to uncalibrated moving cameras through a novel raster bundle adjustment. This research program will lay a computational foundation for two major scientific disciplines: (1) Behavioral science: the raster theory will address many customized visual tasks to characterize microscopic social signals, enabling us to overcome the fundamental limitations of existing approaches in behavioral assessments for at-risk children (such as those with autism spectrum disorder, which have typically been assessed using a subjective and sporadic measure); and (2) Neuroscience: the planned research will enable computational measurement of free-ranging activities of primate subjects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

各种类型的摄像机继续在我们的环境中激增，不断记录我们的日常行为以及与周围物体，场景和其他人的互动。然而，这种收集视觉数据的能力不断增长，却没有与被记录的个人（由自己或他人）按照自己的意愿从这些数据中测量、分析和预测的能力相匹配。该项目开发了一个计算机视觉系统，该系统执行从个人视觉数据中学习的任务，例如，通过用户的可穿戴摄像头进行饮食监测。该项目将开发一种通用的3D几何表示来编码多视图图像流，该图像流可以提供辅助空间约束以实现半监督学习。这种表示将通过利用大量或潜在无限数量的未标记多视图图像流来实现许多计算机视觉任务。研究人员计划通过多种教育活动传播这项研究，包括：（a）通过在夏季技术营，女孩机器学习日营和明尼苏达大学机器人展中举办的一系列讲习班，为代表性不足的K-12学生提供服务;（B）每年在明尼苏达州博览会上公开展示研究成果;（c）透过编订短期课程教材，让本科生参与研究;（d）发展光栅多视代数新课程，以巩固多视几何学和视觉认识（深度学习）;以及（e）通过教程和研讨会组织更广泛的受众在计算机视觉的研究成果的传播。这项研究调查了一个新的光栅表示，多视图几何，提供了三维重建与视觉识别模型训练的紧密集成。该理论将允许以两种方式广泛利用未标记的多视图图像流：（1）通过以无监督的方式通过3D重建主动探索多视图视觉数据，以及（2）通过几何地跨视图传递概率信念以用于跨视图监督。该项目将通过一种新的光栅光束法平差将这一理论推广到未标定的移动相机。该研究计划将为两大科学学科奠定计算基础：（1）行为科学：光栅理论将解决许多定制的视觉任务，以表征微观社会信号，使我们能够克服现有方法在高危儿童行为评估中的基本局限性（例如那些患有自闭症谱系障碍的人，通常使用主观和零星的测量进行评估）;和（2）神经科学：计划中的研究将使灵长类动物自由活动的计算测量成为可能。2该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

OpenMonkeyChallenge: Dataset and Benchmark Challenges for Pose Estimation of Non-human Primates.

• DOI: 10.1007/s11263-022-01698-2
• 发表时间: 2023-01
• 期刊: INTERNATIONAL JOURNAL OF COMPUTER VISION
• 影响因子: 19.5
• 作者: Yao, Yuan;Bala, Praneet;Mohan, Abhiraj;Bliss-Moreau, Eliza;Coleman, Kristine;Freeman, Sienna M.;Machado, Christopher J.;Raper, Jessica;Zimmermann, Jan;Hayden, Benjamin Y.;Park, Hyun Soo
• 通讯作者: Park, Hyun Soo

Multiview Supervision By Registration

• DOI: 10.1109/wacv45572.2020.9093591
• 发表时间: 2018-11
• 期刊: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV)
• 作者: Yilun Zhang;H. Park

Multiview Cross-supervision for Semantic Segmentation

• 发表时间: 2018-12
• 期刊: ArXiv
• 作者: Y. Yao;H. Park

Self-supervised 3D Representation Learning of Dressed Humans from Social Media Videos

从社交媒体视频中对着装人类进行自监督 3D 表示学习

• DOI: 10.1109/tpami.2022.3231558
• 发表时间: 2022
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Jafarian, Yasamin;Park, Hyun Soo
• 通讯作者: Park, Hyun Soo

Normal-guided Garment UV Prediction for Human Re-texturing

• DOI: 10.1109/cvpr52729.2023.00449
• 发表时间: 2023-03
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Yasamin Jafarian;Tuanfeng Y. Wang;Duygu Ceylan;Jimei Yang;N. Carr;Yi Zhou;Hyunjung Park