A dynamic interactive account of human visual action understanding

日常生活充满了与他人的相遇。通常情况下，我们很快就能毫不费力地理解他们所做动作的含义——这是一种非凡的人类能力。这样做是正常社交生活的关键，因为这些行为提供了关于他人意图、信仰和个性的重要线索。例如，当我们看到一个家庭成员在厨房里切菜时，我们知道他打算做饭；看到朋友撑伞，表示她相信马上就要下雨了；观察一个陌生人在商店门口捐款表明她可能是一个善解人意的人。人们对行动理解是如何如此容易地实现的仍知之甚少。我们的项目提供了一种新的观点，即人类行为的理解源于两种心理过程的相互作用。感知系统收集我们所看到的动作的证据，提取构成动作的物体、动作、身体姿势和场景背景。回到之前的烹饪例子，这些系统将定位和识别刀、砧板、蔬菜和其他物体；计算厨师的姿势，他拿刀的姿势，刀的上下动作；描述场景的布局，并确定它是一个厨房。来自这些感知系统的证据与“行动框架”的心理库相互作用，每个“行动框架”都捕捉到构成行动的典型角色、关系和原因。例如，“烹饪”的动作框架捕获了我们的知识，即这通常涉及使用某些工具和动作操纵食物成分，目的是将它们转化为可食用的结果，所有这些通常都发生在厨房环境中。行动框架也表达了我们关于行动相关概率的一些知识（通常是无意识的）。例如，我们知道切菜的动作更可能是用刀而不是勺子；搅拌经常出现在烹饪中，也出现在绘画中；在厨房里发生的各种行为往往与在车库里看到的那些行为不重叠。当感知系统和行动框架的活动汇聚在一致的解释上时，行动理解就产生了，在这种解释中，行动框架的关键角色被填补，而竞争的、不太可能的行动框架被排除在外。我们计划以两种方式测试这个框架。首先，我们设计了简单的与动作相关的任务，这些任务需要成年志愿者的判断——比如注意两个接连出现的动作是否相同；或者判断用书面标签来描述一部动作电影是对还是错。这些测试分为三大主题。简而言之，他们研究了：1)感知信息退化对行动理解的影响；2)期望如何影响行动理解的效率；3)行动框架如何“填充”我们实际上看不到的行动方面。第四个跨领域主题评估心理“负荷”（例如视觉干扰或同时处理多个心理任务）如何影响行动理解。我们的第二种方法是用简单但功能强大的“神经网络”计算机模型对这些任务进行详细建模。这使我们能够以一种精确的、定量的方式来构建我们的预测，并对行动理解行为将如何展开做出新的预测。通过这种结合的方法，我们希望证明我们的框架如何至少部分解释人类有效理解他人行为的能力。我们认为这个框架有潜力为未来儿童发展、群体动力学、社会学习、人工视觉或其他学科的研究提供信息，这些学科与人类观察者如何理解他人行为背后的意义和学习机会有关。我们建议召集来自这些和相关学科的感兴趣的研究人员组成一个国际联盟，以加速这些潜在的影响。

Daily life is filled with encounters with other people. Normally, we quickly and effortlessly understand the meaning of the actions they perform -- a remarkable human capacity. Doing so is key to normal social life, because those actions provide vital clues about others' intentions, beliefs, and personalities. For example, on seeing a family member chopping vegetables in the kitchen, we know that he intends to cook a meal; seeing a friend opening an umbrella suggests that she believes it will soon rain; and observing a stranger make a donation at a shop entrance indicates that she may be an empathetic person. How action understanding is so readily achieved remains poorly understood.Our project offers a novel view of human action understanding as arising from interactions of two mental processes. Perceptual systems gather evidence about the actions we see, extracting the objects, movements, body postures, and scene context that make up an action. Returning to the previous cooking example, these systems would locate and identify the knife, cutting board, vegetables, and other objects; compute the posture of the cook, his grasp of the knife, and its up-and-down movements; and describe the layout of the scene and identify it as a kitchen. Evidence from these perceptual systems interacts with a mental library of "action frames", each of which captures the typical roles, relationships, and reasons that comprise an action. For example, an action frame for "cooking" captures our knowledge that this generally involves the manipulation of food ingredients, with certain tools and movements, with the goal to transform them into an edible result, all of which typically takes place in a kitchen setting. Action frames also express some of our (normally unconscious) knowledge about probabilities related to actions. For example, we know that chopping motions are more likely to occur with a knife than a spoon; stirring often occurs in cooking but also in painting; and the kinds of actions that occur in a kitchen tend not to overlap with those typically seen in a garage. Action understanding arises when the activity of the perceptual systems and the action frames converges on a consistent interpretation, in which the key roles of the action frame are filled, and competing, less-likely action frames are excluded. We plan to test this framework in two ways. First, we have designed simple action-related tasks that will require judgments from human adult volunteers - such as to notice whether two actions that are shown one after the other are the same or not; or to judge whether a written label is the right or wrong one to describe an action picture. These tests are grouped under three broad themes. In brief, they examine: 1) the impact of degraded perceptual information on action understanding; 2) how expectations affect the efficiency of action understanding; and 3) how action frames "fill in" aspects of actions that we don't actually see. A fourth cross-cutting theme assesses how mental "load" (e.g. visual distractions or juggling multiple mental tasks) impacts action understanding. Our second approach is to model each of these tasks in detail with simple but powerful "neural network" computer models. These allow us to frame our predictions in a precise, quantitative way, and to make new predictions about how action understanding behaviour will unfold. With this combined approach, we hope to demonstrate how our framework explains at least some of the human ability to understand others' actions efficiently. We see potential for this framework to inform future research in child development, group dynamics, social learning, artificial vision or other disciplines with a stake in how human observers understand the meaning and learning opportunities behind others' actions. We propose to assemble an international Consortium of interested researchers from these and related disciplines, to accelerate those potential impacts.