UMPIRE: United Model for the Perception of Interactions in visuoauditory REcognition

人类每天在家里(例如做饭/清洁)或户外(例如售票机/购物袋)、在工作期间(例如组装/机械)或休闲时间(例如玩/运动)单独或协作地与数十个物体交互。在观察人与物体的互动时，我们的视觉辅以听觉是感知这些互动的主要工具。让我们以烧开水壶里的水为例。我们观察演员按下按钮，等着听到水开了，茶壶的灯熄灭了，然后水才被用来泡茶。知觉过程是通过理解有意的相互作用(称为意念运动动作)以及对环境中动态刺激的反应动作(称为感觉运动动作)而形成的。作为观察者，我们理解并最终可以使用我们的感官输入以及我们对事件感知的潜在复杂认知过程来复制这种互动。行为科学中的证据表明，这些人类认知过程是高度模块化的，这些模块协作实现了我们出色的人类水平的感知。然而，目前人工智能领域的方法缺乏模块化，因此缺乏能力。为了实现人类对对象交互的感知，包括当交互导致错误(例如，水溢出)或风险(例如，沸水溢出)时的在线感知，该奖学金专注于从经过充分研究的认知行为框架中通知计算机视觉和机器学习模型，包括深度学习体系结构。与其手工制作的前身相比，深度学习体系结构在视频级别分类方面取得了更好的性能，但它们在视频中的细粒度理解方面的表现仍然不是很好。最近的研究表明，当前的模型很容易被类似的动作或不完整的动作所愚弄。该奖学金专注于通过模块化来增强这些模型的能力，这一原则自50年代以来在Fodor的《大脑模块化》中得到了验证，并经常被认知心理学家在受控实验室环境中研究。高层感知的模块化，以及深度学习架构的能力，将为以前未开发的视频分析带来新的理解。对日常和稀有对象交互的有针对性的感知，将为使用可穿戴计算的辅助技术和机器人模仿学习等应用奠定基础。我们将与三个行业伙伴密切合作，为应用程序铺平潜在的知识转移道路。此外，该奖学金将通过研讨会、基准和大型数据集的公开挑战积极吸引国际研究人员，以鼓励其他研究人员解决与视频理解中的细粒度感知相关的问题。

Humans interact with tens of objects daily, at home (e.g. cooking/cleaning) or outdoors (e.g. ticket machines/shopping bags), during working (e.g. assembly/machinery) or leisure hours (e.g. playing/sports), individually or collaboratively. When observing people interacting with objects, our vision assisted by the sense of hearing is the main tool to perceive these interactions. Let's take the example of boiling water from a kettle. We observe the actor press a button, wait and hear the water boil and the kettle's light go off before water is used for, say, preparing tea. The perception process is formed from understanding intentional interactions (called ideomotor actions) as well as reactive actions to dynamic stimuli in the environment (referred to as sensormotor actions). As observers, we understand and can ultimately replicate such interactions using our sensory input, along with our underlying complex cognitive processes of event perception. Evidence in behavioural sciences demonstrates that these human cognitive processes are highly modularised, and these modules collaborate to achieve our outstanding human-level perception.However, current approaches in artificial intelligence are lacking in their modularity and accordingly their capabilities. To achieve human-level perception of object interactions, including online perception when the interaction results in mistakes (e.g. water is spilled) or risks (e.g. boiling water is spilled), this fellowship focuses on informing computer vision and machine learning models, including deep learning architectures, from well-studied cognitive behavioural frameworks.Deep learning architectures have achieved superior performance, compared to their hand-crafted predecessors, on video-level classification, however their performance on fine-grained understanding within the video remains modest. Current models are easily fooled by similar motions or incomplete actions, as shown by recent research. This fellowship focuses on empowering these models through modularisation, a principle proven since the 50s in Fodor's Modularity of the Mind, and frequently studied by cognitive psychologists in controlled lab environments. Modularity of high-level perception, along with the power of deep learning architectures, will bring a new understanding to videos analysis previously unexplored.The targeted perception, of daily and rare object interactions, will lay the foundations for applications including assistive technologies using wearable computing, and robot imitation learning. We will work closely with three industrial partners to pave potential knowledge transfer paths to applications.Additionally, the fellowship will actively engage international researchers through workshops, benchmarks and public challenges on large datasets, to encourage other researchers to address problems related to fine-grained perception in video understanding.