Robot Learning and Discovering Through Memorizing Visual and Auditory Interactions

自成立以来，人工智能(AI)的目标是成为一门系统科学，目标是在狂野、混乱的世界中工作，解决体系结构、学习和评估等关键挑战。为了实现这一目标，人-机器人交互(HRI)通过解决具体化人工智能的集成挑战带来了丰富的机会。我的探索助学金(DG)研究计划的长期目标是研究如何整合所有必要的决策过程，从导航到任务完成以及与其他人(机器人或人类)的互动，以便机器人能够在现实生活中长时间运行。我的战略包括设计具有越来越多的高级感知、推理和动作能力的机器人，这些能力包括实时执行、健壮性、适应性和可伸缩性，能够在现实生活中使用。在短期内，我的研究计划侧重于从时空视觉和听觉模式的使用中学习，使机器人从其操作环境中获取知识。记忆在识别模式和预测即将到来的知觉和行为后果方面发挥着核心作用，这是智能的一个关键特征和关键组成部分。基于我在基于视觉的导航和情节记忆模型方面的工作，以及来自视觉和音频数据的交叉参考信息(使用链接到视觉数据的空间听觉数据)，我将研究如何通过在动态条件下操作和与人互动来提高自主机器人推导概念、预测未来事件和制定行为策略的能力。深度神经网络(DNN)被认为是学习视觉和音频数据复杂性的强大工具，这有助于提供线索和指示，说明机器人正在经历的事情可能是什么有趣的元素。使用设计用于检测对象、人物、人脸、姿势、声音和语音的DNN，我将使用不同的表示/模型来记忆交互历史，从中获取知识和理解。实验包括使用商业上可获得的定制设计的机器人在长期的HRI情景中进行试验。我的研究项目涉及2名博士后、2名博士、2名硕士和2名本科生，集成了从视觉和音频处理到机器人控制架构、记忆模型、决策过程和HRI的大量组件，在视觉-听觉认知和体验式多通道交互的语义解释领域做出了贡献。它的影响范围从工厂装配线上的应用到医疗保健、康复和老龄化以及监测。我所有由DG计划资助的研究贡献都是开源的，允许分享并为让机器人更接近人类、提高生活质量以及我们对设计真正智能机器人的理解和能力的共同努力做出贡献。

Since its foundation, Artificial Intelligence (AI) aims at becoming a system science targeting working in the wild, messy world, addressing key challenges such as architecture, learning and evaluation. To accomplish this, human-robot interaction (HRI) brings rich opportunities by addressing the integration challenges of embodied AI. The long term objective of my Discovery Grant (DG) research program is the study of how to integrate all the required decision-making processes, ranging from navigation to task accomplishment and interaction with others (robots or humans), so that robots can operate in real life settings over long periods of time. My strategy consists of designing robots with an increasing set of advanced perceptual, reasoning and action capabilities under real-time execution, robustness, adaptability and scalability, capable of being used in real life settings. In the short term, my research program focuses on learning from the use of spatial-temporal visual and auditory patterns to make the robots derive knowledge from its operation environments. Memory plays a central role in recognizing patterns and predicting upcoming percepts and action consequences, which is a key feature and critical component of intelligence. Building on my work on vision-based navigation and episodic memory models, along with cross-referenced information from visual and audio data (using spatial auditory data linked to visual data), I will study how to increase the ability of an autonomous robot to derive concepts, predict future events and elaborate behavioral strategies, by operating in dynamic conditions and interacting with people. Deep Neural Networks (DNNs) reveal to be powerful tools to learn the complexities of visual and audio data, which can be beneficial in providing cues and indications of what could be interesting elements in what the robot is experiencing. Using DNNs designed to detect objects, people, faces, pose, sounds and voice, I will use different representations/models to memorize the interaction history from which to derive knowledge and understanding. Experimentation involves conducting trials in long-lasting HRI scenarios using commercially available and custom-designed robots. Involving 2 postdoc, 2 PhDs, 2 Master's and 2 undergraduates, my research program integrates a large set of components, from vision and audio processing to robot control architecture, memory models, decision-making processes and HRI, with contributions to the fields of visuo-auditory cognition and semantic interpretation of embodied multimodal interaction. Its impacts range from applications on factory assembly lines to healthcare, rehabilitation and aging, and surveillance. All my research contributions funded by the DG program are open source, allowing sharing and contributing to the joint effort of bringing robots closer to people, improving quality of life and our understanding and ability of designing truly intelligent robots.