CPS: Synergy: MONA LISA - Monitoring and Assisting with Actions

在不久的将来，网络物理系统将与人类合作。这样的认知系统需要理解人类在做什么。它们需要实时解释人类的行为，并预测人类在复杂、嘈杂和混乱的环境中的直接意图。该提案提出了一种新的认知网络物理系统架构，可以理解复杂的人类活动，并特别关注操纵活动。所提出的架构，生物感知和控制的动机，由三层组成。在底层是视觉过程，检测，识别和跟踪人类，他们的身体部位，物体，工具和物体几何形状。中间层包含人类活动的符号模型，它通过语法描述将前一层识别的信号成分组装成正在进行的活动的表示。最后，在顶层是认知控制，它决定场景的哪些部分将被处理，以及哪些算法将被应用于何处。它通过在需要时获取额外的知识来调节视觉过程，并通过控制主动视觉系统将其传感器引导到特定位置来引导注意力。因此，底层是感知，中间层是认知，顶层是控制。所有层都可以访问离线过程中构建的知识库，其中包含有关操作的语义。该方法的可行性将通过开发一个名为莫纳丽莎的智能制造系统来证明，该系统可帮助人类完成装配任务。该系统将在人类执行装配任务时进行监控。它将识别装配动作并确定其是否正确，并将向人类传达可能的错误并建议继续进行的方法。该系统将具有先进的视觉传感和感知;基于机器人和人类研究的动作理解;语义和类似程序的记忆和推理，以及一个连接高级推理和低级感知的控制模块，用于真实的时间，与人类装配人员的反应和主动参与。拟议的工作将为传感器网络和机器人领域带来新的工具和方法，除了智能制造外，还适用于各种行业和应用。能够使用视觉传感器分析人类行为将对许多领域产生影响，从医疗保健和高级驾驶员辅助到人类机器人协作。该项目还将促进K-12推广，新课件（本科生和研究生），出版物和开源软件。

Cyber-physical systems of the near future will collaborate with humans. Such cognitive systems will need to understand what the humans are doing. They will need to interpret human action in real-time and predict the humans' immediate intention in complex, noisy and cluttered environments. This proposal puts forward a new architecture for cognitive cyber-physical systems that can understand complex human activities, and focuses specifically on manipulation activities. The proposed architecture, motivated by biological perception and control, consists of three layers. At the bottom layer are vision processes that detect, recognize and track humans, their body parts, objects, tools, and object geometry. The middle layer contains symbolic models of the human activity, and it assembles through a grammatical description the recognized signal components of the previous layer into a representation of the ongoing activity. Finally, at the top layer is the cognitive control, which decides which parts of the scene will be processed next and which algorithms will be applied where. It modulates the vision processes by fetching additional knowledge when needed, and directs the attention by controlling the active vision system to direct its sensors to specific places. Thus, the bottom layer is the perception, the middle layer is the cognition, and the top layer is the control. All layers have access to a knowledge base, built in offline processes, which contains the semantics about the actions.The feasibility of the approach will be demonstrated through the development of a smart manufacturing system, called MONA LISA, which assists humans in assembly tasks. This system will monitor humans as they perform assembly task. It will recognize the assembly action and determine whether it is correct and will communicate to the human possible errors and suggest ways to proceed. The system will have advanced visual sensing and perception; action understanding grounded in robotics and human studies; semantic and procedural-like memory and reasoning, and a control module linking high-level reasoning and low-level perception for real time, reactive and proactive engagement with the human assembler. The proposed work will bring new tools and methodology to the areas of sensor networks and robotics and is applicable, besides smart manufacturing, to a large variety of sectors and applications. Being able to analyze human behavior using vision sensors will have impact on many sectors, ranging from healthcare and advanced driver assistance to human robot collaboration. The project will also catalyze K-12 outreach, new courseware (undergraduate and graduate), publication and open-source software.