Structure Induction for Manipulative and Interactive Tasks

操作和交互任务的结构归纳本项目将开发用于推理模型的方法，类似于传统上用于语音和语言的方法，用于结构化、重复、交互式操作任务。这些任务通常由更基本的任务组成，这些任务遵循一个名义上的行动计划，以实现一组预期的结果。例子包括组装任务、各种形式的手术、基于手势的界面、交通操纵、运动、舞蹈或体操等等。该项目将在微创外科手术过程的观察和建模的背景下进行。在使用达芬奇手术机器人模拟手术任务时，将从外科医生那里获得运动和视频流。手术任务将使用语法在一组简单的、基本的手术手势上表示，这些手势使用有限状态机建模。基本手势和任务相关的语法约束都将直接从数据中推断出来。该方法的一个关键要素是首先从简单的低维数据集（如手术运动的直接测量）开发准确的模型，然后使用这些模型将模型训练引导到更复杂的高维同步视频中。因此，需要解决的核心研究问题是：基本任务的运动序列建模：开发能够学习有限状态机的算法，该算法使用有限的手动标记运动数据集，根据原子手势建模简单的，与上下文无关的基本运动或操作任务。基本任务的多模态模型：创建有效的方法，用来自同步视频数据的互补和上下文特征来增强基本任务模型。语法归纳：使用基本任务模型自动标记更多、更大的数据序列，并从这些标记中推断高级语法结构。模型引导：使用基本任务和语法模型来标记更大的数据公司，然后仅使用视频数据来重新训练基本任务模型。总之，本项目将研究结构化人类操作动作的建模问题，例如在微创手术中发生的操作动作。使用最初为语音理解而开发的统计建模方法，将创建和评估手术动作语言的模型。这些模型将首先使用手术机器人的运动测量来创建，然后将从同一系统传输到视频数据。这些特定的模型将适用于外科训练、技能评估和机器人手术增强。更一般地说，该项目将提高我们开发用于理解复杂数据源的结构化模型的能力，这具有广泛的意义。它将促进各种需要理解人类行为的应用，如用于人机交互的基于手势的系统，用于解释或检测视频监控数据中的动作的方法，或用于生物力学和医学的运动评估和测量。与此同时，它将推进大型复杂数据流（如视频、文本、语音或医学图像）建模的基础科学。URL: http://www.cs.jhu.edu/CIRL/projects/SurgicalModeling/

Structure Induction for Manipulative and Interactive TasksThis project will develop methods for inferring models, similar to those traditionally used for speech and language, for structured, repetitive, interactive manipulation tasks. Such tasks are typically composed of more elementary tasks that follow a nominal plan of action to achieve a set of desired outcomes. Examples include assembly tasks, various forms of surgery, gesture-based interfaces, maneuvering in traffic, sports, dancing or gymnastics to name a few. The project will be carried out in the context of observing and modeling minimally invasive surgical procedures. Motion and video streams will be acquired from surgeons during simulated surgical tasks using the da Vinci surgical robot. Surgical tasks will be represented using grammars on a set of simple, elementary surgical gestures modeled using finite-state machines. Both the elementary gestures and the task-dependent grammatical constraints will be inferred directly from data. A key element of the proposed approach is to first develop accurate models from simple, low-dimensional datasets such as direct measurements of surgical motion, and then to use these models to boostrap model training to more complex, high-dimensional synchronized video of the same procedure.The core research problems to be addressed are thus: Motion sequence modeling for elementary tasks: Develop algorithms that are able to learn finite-state machines modeling simple, context-independent elementary motion or manipulation tasks in terms of atomic gestures using a limited set of manually labeled motion data. Multi-modal models of elementary tasks: Create effective methods for augmenting elementary task models with complementary and contextual features from synchronized video data. Grammar induction: Use elementary task models to automatically label more, larger data sequences and infer high-level grammatical structures from these labelings. Model bootstrapping: Use elementary task and grammatical models to label even larger data corpi that are then used to retrain elemetary task models using only video data.In summary, this project will study the problem of modeling structured human manipulative actions such as those that occur during minimally invasive surgery. Using statistical modeling methods originally developed for speech understanding, models for the language of surgical motion will be created and evaluated. Thesemodels will first be created using motion measurements from a surgical robot, but will then be transferred to video data from the same system. These specific models will have applicability to surgical training, skill assessment, and robotic surgical augmentation. More generally, this project will advance our ability to develop structured models for understanding com-plex data sources, which has wide ranging implications. It will facilitate a variety of applications that require understanding of human action such gesture-based systems for human-machine interaction, methods for interpreting or detecting action in video surveillance data, or assessment and measurement of movement for bio-mechanics and medicine. At the same time, it will advance the basic science of modeling large and complex data streams such as video, text, speech, or medical images.URL: http://www.cs.jhu.edu/CIRL/projects/SurgicalModeling/