Robot Learning and Discovering Through Memorizing Visual and Auditory Interactions

点击翻译按钮获取中文摘要

Since its foundation, Artificial Intelligence (AI) aims at becoming a system science targeting working in the wild, messy world, addressing key challenges such as architecture, learning and evaluation. To accomplish this, human-robot interaction (HRI) brings rich opportunities by addressing the integration challenges of embodied AI. The long term objective of my Discovery Grant (DG) research program is the study of how to integrate all the required decision-making processes, ranging from navigation to task accomplishment and interaction with others (robots or humans), so that robots can operate in real life settings over long periods of time. My strategy consists of designing robots with an increasing set of advanced perceptual, reasoning and action capabilities under real-time execution, robustness, adaptability and scalability, capable of being used in real life settings. In the short term, my research program focuses on learning from the use of spatial-temporal visual and auditory patterns to make the robots derive knowledge from its operation environments. Memory plays a central role in recognizing patterns and predicting upcoming percepts and action consequences, which is a key feature and critical component of intelligence. Building on my work on vision-based navigation and episodic memory models, along with cross-referenced information from visual and audio data (using spatial auditory data linked to visual data), I will study how to increase the ability of an autonomous robot to derive concepts, predict future events and elaborate behavioral strategies, by operating in dynamic conditions and interacting with people. Deep Neural Networks (DNNs) reveal to be powerful tools to learn the complexities of visual and audio data, which can be beneficial in providing cues and indications of what could be interesting elements in what the robot is experiencing. Using DNNs designed to detect objects, people, faces, pose, sounds and voice, I will use different representations/models to memorize the interaction history from which to derive knowledge and understanding. Experimentation involves conducting trials in long-lasting HRI scenarios using commercially available and custom-designed robots. Involving 2 postdoc, 2 PhDs, 2 Master's and 2 undergraduates, my research program integrates a large set of components, from vision and audio processing to robot control architecture, memory models, decision-making processes and HRI, with contributions to the fields of visuo-auditory cognition and semantic interpretation of embodied multimodal interaction. Its impacts range from applications on factory assembly lines to healthcare, rehabilitation and aging, and surveillance. All my research contributions funded by the DG program are open source, allowing sharing and contributing to the joint effort of bringing robots closer to people, improving quality of life and our understanding and ability of designing truly intelligent robots.