CPS: Synergy: MONA LISA - Monitoring and Assisting with Actions

不久的将来，网络物理系统将与人类合作。这样的认知系统将需要理解人类正在做什么。他们将需要实时解释人类的行为，并预测人类在复杂、嘈杂和杂乱的环境中的直接意图。这一建议提出了一种新的认知网络物理系统架构，它可以理解复杂的人类活动，并特别关注操纵活动。建议的体系结构由三个层次组成，由生物感知和控制驱动。最底层是检测、识别和跟踪人类及其身体部位、物体、工具和物体几何形状的视觉过程。中间层包含人类活动的符号模型，它通过语法描述将前一层识别的信号成分组装成正在进行的活动的表示。最后，在顶层是认知控制，它决定下一步处理场景的哪些部分，以及在哪里应用哪些算法。它通过在需要时获取额外的知识来调节视觉过程，并通过控制主动视觉系统将传感器引导到特定位置来引导注意力。因此，底层是感知，中层是认知，顶层是控制。所有层都可以访问一个内置在离线流程中的知识库，其中包含有关动作的语义。该方法的可行性将通过开发一个名为MONA LISA的智能制造系统来演示，该系统帮助人类完成装配任务。该系统将在人类执行装配任务时对其进行监控。它将识别装配动作并确定其是否正确，并将向人类传达可能的错误并建议继续进行的方法。该系统将拥有先进的视觉感知和感知；基于机器人学和人体研究的行动理解；语义和程序性记忆和推理；以及一个连接高级推理和低级别感知的控制模块，用于与人类组装者进行实时、被动和主动的接触。拟议的工作将为传感器网络和机器人领域带来新的工具和方法，除智能制造外，还适用于各种部门和应用。能够使用视觉传感器分析人类行为将对许多领域产生影响，从医疗保健和先进的驾驶员辅助到人类机器人协作。该项目还将促进K-12的推广、新的课程软件(本科生和研究生)、出版物和开放源码软件。

Cyber-physical systems of the near future will collaborate with humans. Such cognitive systems will need to understand what the humans are doing. They will need to interpret human action in real-time and predict the humans' immediate intention in complex, noisy and cluttered environments. This proposal puts forward a new architecture for cognitive cyber-physical systems that can understand complex human activities, and focuses specifically on manipulation activities. The proposed architecture, motivated by biological perception and control, consists of three layers. At the bottom layer are vision processes that detect, recognize and track humans, their body parts, objects, tools, and object geometry. The middle layer contains symbolic models of the human activity, and it assembles through a grammatical description the recognized signal components of the previous layer into a representation of the ongoing activity. Finally, at the top layer is the cognitive control, which decides which parts of the scene will be processed next and which algorithms will be applied where. It modulates the vision processes by fetching additional knowledge when needed, and directs the attention by controlling the active vision system to direct its sensors to specific places. Thus, the bottom layer is the perception, the middle layer is the cognition, and the top layer is the control. All layers have access to a knowledge base, built in offline processes, which contains the semantics about the actions.The feasibility of the approach will be demonstrated through the development of a smart manufacturing system, called MONA LISA, which assists humans in assembly tasks. This system will monitor humans as they perform assembly task. It will recognize the assembly action and determine whether it is correct and will communicate to the human possible errors and suggest ways to proceed. The system will have advanced visual sensing and perception; action understanding grounded in robotics and human studies; semantic and procedural-like memory and reasoning, and a control module linking high-level reasoning and low-level perception for real time, reactive and proactive engagement with the human assembler. The proposed work will bring new tools and methodology to the areas of sensor networks and robotics and is applicable, besides smart manufacturing, to a large variety of sectors and applications. Being able to analyze human behavior using vision sensors will have impact on many sectors, ranging from healthcare and advanced driver assistance to human robot collaboration. The project will also catalyze K-12 outreach, new courseware (undergraduate and graduate), publication and open-source software.