Robot Learning and Discovering Through Memorizing Visual and Auditory Interactions

机器人通过记忆视觉和听觉交互来学习和发现

• 批准号: RGPIN-2022-04036
• 负责人: Michaud, François
• 金额: $ 3.5万
• 依托单位: Université de Sherbrooke
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=754136
• 关键词: Robot; Learning; Discovering; Through; Memorizing

Since its foundation, Artificial Intelligence (AI) aims at becoming a system science targeting working in the wild, messy world, addressing key challenges such as architecture, learning and evaluation. To accomplish this, human-robot interaction (HRI) brings rich opportunities by addressing the integration challenges of embodied AI. The long term objective of my Discovery Grant (DG) research program is the study of how to integrate all the required decision-making processes, ranging from navigation to task accomplishment and interaction with others (robots or humans), so that robots can operate in real life settings over long periods of time. My strategy consists of designing robots with an increasing set of advanced perceptual, reasoning and action capabilities under real-time execution, robustness, adaptability and scalability, capable of being used in real life settings. In the short term, my research program focuses on learning from the use of spatial-temporal visual and auditory patterns to make the robots derive knowledge from its operation environments. Memory plays a central role in recognizing patterns and predicting upcoming percepts and action consequences, which is a key feature and critical component of intelligence. Building on my work on vision-based navigation and episodic memory models, along with cross-referenced information from visual and audio data (using spatial auditory data linked to visual data), I will study how to increase the ability of an autonomous robot to derive concepts, predict future events and elaborate behavioral strategies, by operating in dynamic conditions and interacting with people. Deep Neural Networks (DNNs) reveal to be powerful tools to learn the complexities of visual and audio data, which can be beneficial in providing cues and indications of what could be interesting elements in what the robot is experiencing. Using DNNs designed to detect objects, people, faces, pose, sounds and voice, I will use different representations/models to memorize the interaction history from which to derive knowledge and understanding. Experimentation involves conducting trials in long-lasting HRI scenarios using commercially available and custom-designed robots. Involving 2 postdoc, 2 PhDs, 2 Master's and 2 undergraduates, my research program integrates a large set of components, from vision and audio processing to robot control architecture, memory models, decision-making processes and HRI, with contributions to the fields of visuo-auditory cognition and semantic interpretation of embodied multimodal interaction. Its impacts range from applications on factory assembly lines to healthcare, rehabilitation and aging, and surveillance. All my research contributions funded by the DG program are open source, allowing sharing and contributing to the joint effort of bringing robots closer to people, improving quality of life and our understanding and ability of designing truly intelligent robots.

自成立以来，人工智能（AI）的目标是成为一门系统科学，目标是在野外，混乱的世界中工作，解决架构，学习和评估等关键挑战。为了实现这一目标，人机交互（HRI）通过解决嵌入式AI的集成挑战带来了丰富的机会。 我的发现补助金（DG）研究计划的长期目标是研究如何整合所有必要的决策过程，从导航到任务完成以及与他人（机器人或人类）的互动，以便机器人可以在真实的生活环境中长时间运行。我的策略包括设计机器人，使其具有越来越多的高级感知，推理和实时执行，鲁棒性，适应性和可扩展性下的行动能力，能够在真实的生活环境中使用。 在短期内，我的研究计划集中在学习使用时空视觉和听觉模式，使机器人从其操作环境中获取知识。记忆在识别模式和预测即将到来的感知和行动后果方面发挥着核心作用，这是智力的关键特征和关键组成部分。基于我在基于视觉的导航和情景记忆模型方面的工作，沿着来自视觉和音频数据的交叉引用信息（使用与视觉数据相关联的空间听觉数据），我将研究如何通过在动态条件下操作并与人互动来提高自主机器人推导概念、预测未来事件和制定行为策略的能力。 深度神经网络（DNN）是学习视觉和音频数据复杂性的强大工具，这有助于提供机器人正在经历的有趣元素的线索和指示。使用DNN来检测物体，人，面孔，姿势，声音和声音，我将使用不同的表示/模型来记住交互历史，从中获得知识和理解。实验包括使用商业上可用的和定制设计的机器人在长期HRI场景中进行试验。 我的研究项目涉及2名博士后，2名博士，2名硕士和2名本科生，从视觉和音频处理到机器人控制架构，记忆模型，决策过程和HRI，我的研究项目集成了大量的组件，对视听认知和体现多模态交互的语义解释领域做出了贡献。其影响范围从工厂装配线到医疗保健、康复和老龄化以及监控。我所有由DG计划资助的研究贡献都是开源的，允许共享和促进共同努力，使机器人更接近人类，提高生活质量和我们设计真正智能机器人的理解和能力。