Collaborative Research: RI: Medium: Robust Perception through End-User Adaptation

为了让智能系统（如机器人或自动驾驶汽车）以安全可靠的方式进入最终用户的日常生活，系统必须超越开发实验室，推广到部署它的不受控制的环境。例如，自动驾驶汽车可以针对阳光明媚的天气进行构建和优化，但可以由用户在结冰或下雪的条件下驾驶。机器学习和感知领域的最新技术无法概括或适应这种部署场景的绝对多样性。这个项目试图通过利用人们是习惯的生物，并且倾向于以特定的方式持续和重复地使用他们的设备（例如，他们在家庭，办公室和市场之间的一小部分路线上反复驾驶汽车）来解决这一挑战。这种重复使用为智能系统提供了充分的机会来适应最终用户的特定环境，无论它们多么具有挑战性或不同。该项目基于这一洞察力，设计出强大的感知系统，以适应各种现实世界的挑战性环境，包括在不同驾驶地点以及各种时间和天气条件下的自动驾驶汽车。确保智能系统能够在如此多样化的环境中可靠地运行，对于释放机器学习、计算机视觉和机器人技术研究人员正在努力实现的社会效益是必要的。除了研究界，该项目还将通过培训本科生和研究生，以及通过研讨会和暑期项目向高中生推广，特别是使代表性不足的少数民族受益，为教育做出贡献。该研究项目通过利用终端用户的特定和众所周知的属性，通过适应来研究强大的感知系统的设计和开发：人类是有习惯的生物，倾向于以特定的方式和环境反复和一致地操作设备。例如，大多数人每天主要沿着沿着相同的路线驾驶他们的汽车。特别是，研究人员探索了三个关键思想：（1）通过记录使用过程中的感觉输入来适应感知系统，生成高度可靠的伪标签注释，其中包含物理约束和跨传感器一致性，并通过双任务协同适应在离线时微调系统;（2）通过重复使系统个性化，通过随时间调整回放以利用深度神经网络的记忆能力，并通过在记录之间进行标签传播来增强不同设置的数据;（3）通过开发使用学习动态和主动用户验证来检测和去除噪声标签的方法来验证适应性。这三个研究目标将由一个全面的评估计划来补充，该计划将包括多个现有的自动驾驶数据集，一个由研究人员团队新收集的数据集，该数据集将捕获沿着重复路线的不同环境，以及家用机器人场景中的导航。这项针对更大，更具挑战性的适应问题的研究工作将为计算机视觉，机器学习和机器人技术的交叉领域打开新的解决方案的大门，包括但不限于物理推理，对丰富的感知任务之间的关系建模，适应不断变化的输出分布，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响进行评估来支持审查标准。

For an intelligent system (such as a robot or a self-driving car) to enter end-users' daily lives in a safe and reliable way, the system must generalize beyond the development laboratory to uncontrolled environments where it will be deployed. For instance, a self-driving car may be built and optimized for sunny weather but may be driven by the user in icy or snowy conditions. The state of the art in machine learning and perception cannot generalize or adapt to this sheer diversity of deployment scenarios. This project seeks to address this challenge by leveraging the fact that people are creatures of habit and tend to use their devices consistently and repeatedly in specific ways (for example, they drive their cars repeatedly over a small set of routes between their home, office, and the marketplace). Such repetitive usage provides ample opportunities for the intelligent system to adapt itself to the end-user's specific circumstances, no matter how challenging or different they are. This project builds upon this insight to design robust perceptual systems that will adapt to a diverse array of real-world challenging settings, including self-driving cars in different driving locations and various time and weather conditions. Guaranteeing that an intelligent system can operate reliably across such diverse settings is necessary to unlock the societal benefits that researchers in machine learning, computer vision, and robotics are striving to achieve. Beyond the research community, the project will contribute to education by training undergraduate and graduate students and by outreach to high-school students through workshops and summer programs, especially to benefit underrepresented minorities.This research project investigates the design and development of robust perceptual systems through adaptation, by exploiting a specific and well-known property of end-users: Humans are creatures of habit and tend to operate devices in specific ways and environments repeatedly and consistently. For example, most people drive their cars primarily along the same routes every day. In particular, the investigators explore three key ideas: (1) adapting the perceptual system by recording sensory input during usage, generating highly reliable pseudo-label annotations that incorporate physical constraints and cross-sensor consistency, and fine-tuning the system while it is offline via dual-task co-adaptation; (2) personalizing the system through repetition, by aligning playbacks over time to leverage deep neural networks' ability to memorize and by augmenting data for diverse settings through label propagation across recordings; (3) verifying adaptation by developing methods to detect and remove noisy labels using learning dynamics and active user verification. These three research aims will be complemented by a comprehensive evaluation plan to include multiple existing self-driving data sets, a newly collected data set by the team of investigators that captures diverse environments along a repeated route, and navigation in home robot scenarios. This research effort towards a much larger, more challenging adaptation problem will open the door to novel solutions in the intersection of computer vision, machine learning, and robotics, including but not limited to reasoning about physics, modeling the relationships between rich perceptual tasks, adapting to changing output distributions, and leveraging patterns in the provenance of the data itself.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.