Adaptive cognition for automated sports video annotation (ACASVA)

开发一种能够自主理解和解释现实世界事件模式的机器仍然是人工智能领域一个具有挑战性的目标。人类能够通过发展复杂的物体和事件的内部表征结构以及连接它们的语法来实现这一点。ACASVA旨在研究视觉和语言语法在学习中的相互作用，通过在一个场景中开发语法，在这个场景中，不同事件的数量受到一套规则的约束，被限制为很小：一项运动。我们将分析一场比赛（例如网球）的视频片段，并使用计算机视觉技术逐步将其理解为一系列（可能重叠的）事件，并建立事件语法。我们将对游戏的评论进行类似的音频/语言分析。这两种语法都将用于构建理解游戏的表征结构。视觉表示还受到游戏规则推理的约束，使得对象分类机制优先调谐到游戏相关实体，如“玩家”，而不是游戏无关实体，如“众成员”。我们还将研究视觉和声音这两种模式在学习过程中如何相互影响;视频的解释受到语言语法的影响，反之亦然。此外，当来自视频模式的语法用于影响音频识别时，模式的这种耦合将导致音频和视频事件的改进识别，反之亦然。ACASVA的心理学部分相应地试图了解这些能力是如何在人类中发展的;视觉语法如何在学习问题中组织和使用，这些语法如何被该领域的先前语言知识修改，视觉语法如何映射到语言语法，以及游戏规则推理如何影响低水平的视觉学习（通过凝视行为确定）。这些结果将反馈到机器学习问题，反之亦然，以及为系统提供一个性能基准ACASVA的潜在受益者（除了科学和工程领域的知识受益者）包括广播和在线视频搜索行业。

The development of a machine that can autonomously understand and interpret patterns of real-world events remains a challenging goal in AI. Humans are able to achieve this by developing sophisticated internal representational structures for object and events and the grammars that connect them. ACASVA aims to investigate the interaction between visual and linguistic grammars in learning by developing grammars in a scenario where the number of different events is constrained, by a set of rules, to be small: a sport. We will analyse video footage of a game (e.g. tennis) and use computer vision techniques to progressively understand it as a sequence of (possibly overlapping) events, and build a grammar of events. We will do a similar audio/linguistic analysis on the commentary on the game. Both of these grammars will be used to build a representational structure for understanding the game. Visual representations are additionally constrained by the inference of game rules so that object-classification mechanisms are preferentially tuned to game-relevant entities like 'player' rather than game-irrelevant entities like 'crowd-member'. We will also investigate how the two modes, sight and sound, can influence each other in the learning process; interpretation of the video is affected by the linguistic grammar and vice versa. Furthermore, this coupling of modes will lead to improved recognition of both audio and video events when the grammars from the video modes are used to influence the audio recognition, and vice versa. The psychological component of the ACASVA correspondingly attempts to learn how these capabilities are developed in humans; how visual grammars are organized and employed in the learning problem, how these grammars are modified by prior linguistic knowledge of the domain, how visual grammars map onto linguistic grammars, and how game rule-inferences influence lower-level visual learning (determined via gaze-behaviour). These results will feedback into the machine-learning problem and vice versa, as well as providing a performance benchmark for the system.Potential beneficiaries of ACASVA (in addition to the knowledge beneficiaries within the fields of science and engineering) include the broadcasting and on-line video search industries.