Collaborative Research: RI: Medium: Robust Perception through End-User Adaptation

为了使智能系统（如机器人或自动驾驶汽车）以安全可靠的方式进入最终用户的日常生活，系统必须从开发实验室扩展到将被部署的不受控制的环境。例如，一辆自动驾驶汽车可能是为晴朗的天气而制造和优化的，但可能是在结冰或下雪的条件下由用户驾驶的。机器学习和感知方面的最新技术无法概括或适应这种部署场景的多样性。这个项目试图通过利用这样一个事实来解决这个挑战，即人们是习惯的生物，倾向于以特定的方式持续和重复地使用他们的设备（例如，他们在他们的家、办公室和市场之间的一小段路线上反复驾驶他们的汽车）。这种重复使用为智能系统提供了充分的机会来适应最终用户的具体情况，无论这些情况多么具有挑战性或不同。该项目建立在这一见解的基础上，设计了强大的感知系统，该系统将适应现实世界中各种具有挑战性的环境，包括不同驾驶地点、不同时间和天气条件下的自动驾驶汽车。为了实现机器学习、计算机视觉和机器人领域的研究人员正在努力实现的社会效益，有必要确保智能系统能够在如此多样化的环境中可靠地运行。在研究界之外，该项目还将通过培训本科生和研究生，并通过研讨会和暑期项目向高中生推广，特别是为未被充分代表的少数民族提供帮助，从而为教育做出贡献。这个研究项目通过利用终端用户的一个特定的众所周知的特性，通过适应性来研究健壮的感知系统的设计和开发：人类是习惯的生物，倾向于在特定的方式和环境中重复和一致地操作设备。例如，大多数人每天开车主要沿着相同的路线行驶。研究人员特别探讨了三个关键思想：(1)通过在使用过程中记录感官输入来适应感知系统，生成包含物理约束和跨传感器一致性的高可靠伪标签注释，并通过双任务协同适应在离线时对系统进行微调；(2)通过重复来个性化系统，随着时间的推移调整回放，利用深度神经网络的记忆能力，并通过在录音中传播标签来增加不同设置的数据；(3)通过开发使用学习动态和主动用户验证来检测和去除噪声标签的方法来验证适应性。这三个研究目标将辅以综合评估计划，包括多个现有的自动驾驶数据集、研究小组新收集的沿着重复路线捕获不同环境的数据集，以及家庭机器人场景中的导航。这项针对更大，更具挑战性的适应问题的研究工作将为计算机视觉，机器学习和机器人技术交叉领域的新解决方案打开大门，包括但不限于物理推理，丰富感知任务之间的关系建模，适应变化的输出分布，以及利用数据本身来源的模式。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

For an intelligent system (such as a robot or a self-driving car) to enter end-users' daily lives in a safe and reliable way, the system must generalize beyond the development laboratory to uncontrolled environments where it will be deployed. For instance, a self-driving car may be built and optimized for sunny weather but may be driven by the user in icy or snowy conditions. The state of the art in machine learning and perception cannot generalize or adapt to this sheer diversity of deployment scenarios. This project seeks to address this challenge by leveraging the fact that people are creatures of habit and tend to use their devices consistently and repeatedly in specific ways (for example, they drive their cars repeatedly over a small set of routes between their home, office, and the marketplace). Such repetitive usage provides ample opportunities for the intelligent system to adapt itself to the end-user's specific circumstances, no matter how challenging or different they are. This project builds upon this insight to design robust perceptual systems that will adapt to a diverse array of real-world challenging settings, including self-driving cars in different driving locations and various time and weather conditions. Guaranteeing that an intelligent system can operate reliably across such diverse settings is necessary to unlock the societal benefits that researchers in machine learning, computer vision, and robotics are striving to achieve. Beyond the research community, the project will contribute to education by training undergraduate and graduate students and by outreach to high-school students through workshops and summer programs, especially to benefit underrepresented minorities.This research project investigates the design and development of robust perceptual systems through adaptation, by exploiting a specific and well-known property of end-users: Humans are creatures of habit and tend to operate devices in specific ways and environments repeatedly and consistently. For example, most people drive their cars primarily along the same routes every day. In particular, the investigators explore three key ideas: (1) adapting the perceptual system by recording sensory input during usage, generating highly reliable pseudo-label annotations that incorporate physical constraints and cross-sensor consistency, and fine-tuning the system while it is offline via dual-task co-adaptation; (2) personalizing the system through repetition, by aligning playbacks over time to leverage deep neural networks' ability to memorize and by augmenting data for diverse settings through label propagation across recordings; (3) verifying adaptation by developing methods to detect and remove noisy labels using learning dynamics and active user verification. These three research aims will be complemented by a comprehensive evaluation plan to include multiple existing self-driving data sets, a newly collected data set by the team of investigators that captures diverse environments along a repeated route, and navigation in home robot scenarios. This research effort towards a much larger, more challenging adaptation problem will open the door to novel solutions in the intersection of computer vision, machine learning, and robotics, including but not limited to reasoning about physics, modeling the relationships between rich perceptual tasks, adapting to changing output distributions, and leveraging patterns in the provenance of the data itself.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.