Collaborative Research: RI: Medium: Robust Perception through End-User Adaptation

为了让智能系统(如机器人或自动驾驶汽车)以安全可靠的方式进入最终用户的日常生活，该系统必须超越开发实验室，推广到将部署它的不受控制的环境。例如，自动驾驶汽车可能是为晴朗天气制造和优化的，但可能是由用户在结冰或下雪的条件下驾驶的。机器学习和感知的最新水平不能概括或适应这种纯粹的部署场景的多样性。这个项目试图通过利用这样一个事实来应对这一挑战：人是习惯性的生物，往往以特定的方式一致和重复地使用他们的设备(例如，他们在家、办公室和市场之间的一小部分路线上反复驾驶汽车)。这种重复使用为智能系统提供了充分的机会来适应最终用户的特定环境，无论它们是多么具有挑战性或不同。该项目以这种洞察力为基础，设计强大的感知系统，以适应各种现实世界具有挑战性的环境，包括不同驾驶地点和不同时间和天气条件下的自动驾驶汽车。确保智能系统能够在如此多样化的环境中可靠地运行，对于释放机器学习、计算机视觉和机器人领域的研究人员正在努力实现的社会效益是必要的。在研究界之外，该项目将通过培训本科生和研究生以及通过研讨会和暑期项目接触高中生来促进教育，特别是让代表不足的少数民族受益。该研究项目通过利用最终用户的一个特定和众所周知的属性，通过适应来调查健壮感知系统的设计和开发：人类是习惯的生物，倾向于以特定的方式和环境反复一致地操作设备。例如，大多数人每天主要沿着相同的路线驾驶汽车。特别是，研究人员探索了三个关键想法：(1)通过记录使用过程中的感觉输入来适应感知系统，生成结合了物理约束和跨传感器一致性的高度可靠的伪标签注释，并通过双任务共同适应在系统离线时对其进行微调；(2)通过重复使系统个性化，通过随时间调整回放来利用深层神经网络的记忆能力，通过通过跨录音的标签传播来增强不同设置的数据；(3)通过开发使用学习动力学和主动用户验证来检测和移除噪音标签的方法来验证适应性。这三个研究目标将得到一项全面评估计划的补充，该计划将包括多个现有的自动驾驶数据集、调查团队新收集的捕捉一条重复路线上的不同环境的数据集，以及家用机器人场景中的导航。这项针对更大、更具挑战性的适应问题的研究工作将为计算机视觉、机器学习和机器人学的交叉打开新解决方案的大门，包括但不限于关于物理的推理、对丰富感知任务之间的关系建模、适应不断变化的输出分布以及利用数据本身来源中的模式。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

For an intelligent system (such as a robot or a self-driving car) to enter end-users' daily lives in a safe and reliable way, the system must generalize beyond the development laboratory to uncontrolled environments where it will be deployed. For instance, a self-driving car may be built and optimized for sunny weather but may be driven by the user in icy or snowy conditions. The state of the art in machine learning and perception cannot generalize or adapt to this sheer diversity of deployment scenarios. This project seeks to address this challenge by leveraging the fact that people are creatures of habit and tend to use their devices consistently and repeatedly in specific ways (for example, they drive their cars repeatedly over a small set of routes between their home, office, and the marketplace). Such repetitive usage provides ample opportunities for the intelligent system to adapt itself to the end-user's specific circumstances, no matter how challenging or different they are. This project builds upon this insight to design robust perceptual systems that will adapt to a diverse array of real-world challenging settings, including self-driving cars in different driving locations and various time and weather conditions. Guaranteeing that an intelligent system can operate reliably across such diverse settings is necessary to unlock the societal benefits that researchers in machine learning, computer vision, and robotics are striving to achieve. Beyond the research community, the project will contribute to education by training undergraduate and graduate students and by outreach to high-school students through workshops and summer programs, especially to benefit underrepresented minorities.This research project investigates the design and development of robust perceptual systems through adaptation, by exploiting a specific and well-known property of end-users: Humans are creatures of habit and tend to operate devices in specific ways and environments repeatedly and consistently. For example, most people drive their cars primarily along the same routes every day. In particular, the investigators explore three key ideas: (1) adapting the perceptual system by recording sensory input during usage, generating highly reliable pseudo-label annotations that incorporate physical constraints and cross-sensor consistency, and fine-tuning the system while it is offline via dual-task co-adaptation; (2) personalizing the system through repetition, by aligning playbacks over time to leverage deep neural networks' ability to memorize and by augmenting data for diverse settings through label propagation across recordings; (3) verifying adaptation by developing methods to detect and remove noisy labels using learning dynamics and active user verification. These three research aims will be complemented by a comprehensive evaluation plan to include multiple existing self-driving data sets, a newly collected data set by the team of investigators that captures diverse environments along a repeated route, and navigation in home robot scenarios. This research effort towards a much larger, more challenging adaptation problem will open the door to novel solutions in the intersection of computer vision, machine learning, and robotics, including but not limited to reasoning about physics, modeling the relationships between rich perceptual tasks, adapting to changing output distributions, and leveraging patterns in the provenance of the data itself.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.