Structure Induction for Manipulative and Interactive Tasks

操作和交互任务的结构归纳这个项目将开发推理模型的方法，类似于传统上用于语音和语言的方法，用于结构化、重复、交互的操作任务。这类任务通常由更基本的任务组成，这些任务遵循名义上的行动计划，以实现一系列预期的结果。例如，组装任务、各种形式的手术、基于手势的界面、交通中的机动、体育、舞蹈或体操等等。该项目将在观察和模拟微创外科手术过程的背景下进行。在使用达芬奇手术机器人执行模拟手术任务期间，将从外科医生那里获取运动和视频流。手术任务将在一组简单的、基本的手术手势上使用语法来表示，这些手势使用有限状态机建模。基本手势和依赖于任务的语法约束都将直接从数据中推断出来。提出的方法的一个关键要素是首先从简单的低维数据集(如手术运动的直接测量)建立准确的模型，然后使用这些模型将模型训练提升到更复杂的、高维的相同过程的同步视频。因此，需要解决的核心研究问题是：基本任务的运动序列建模：开发能够学习有限状态机的算法，该算法能够使用有限的人工标记的运动数据集根据原子手势来建模简单的、上下文无关的基本运动或操纵任务。基本任务的多模式模型：创建有效的方法，从同步的视频数据中增加具有互补和上下文特征的基本任务模型。语法归纳：使用基本任务模型自动标记更多、更大的数据序列，并从这些标记中推断高级语法结构。模型引导：使用基本任务和语法模型来标记更大的数据集，然后使用这些数据集来重新训练仅使用视频数据的基本任务模型。综上所述，该项目将研究对结构化的人类操纵行为进行建模的问题，例如在微创手术期间发生的那些行为。使用最初为语音理解开发的统计建模方法，将创建和评估外科运动语言的模型。这些模型将首先使用手术机器人的运动测量来创建，但随后将被传输到来自同一系统的视频数据。这些特定的模型将适用于外科培训、技能评估和机器人手术增强。更广泛地说，这个项目将提高我们开发用于理解复杂数据源的结构化模型的能力，这具有广泛的影响。它将促进各种需要了解人类行动的应用，例如用于人机交互的基于手势的系统，用于解释或检测视频监控数据中的行动的方法，或用于生物力学和医学的运动评估和测量。同时，它将推进对视频、文本、语音或医学图像等大型复杂数据流进行建模的基础科学。网址：http://www.cs.jhu.edu/CIRL/projects/SurgicalModeling/

Structure Induction for Manipulative and Interactive TasksThis project will develop methods for inferring models, similar to those traditionally used for speech and language, for structured, repetitive, interactive manipulation tasks. Such tasks are typically composed of more elementary tasks that follow a nominal plan of action to achieve a set of desired outcomes. Examples include assembly tasks, various forms of surgery, gesture-based interfaces, maneuvering in traffic, sports, dancing or gymnastics to name a few. The project will be carried out in the context of observing and modeling minimally invasive surgical procedures. Motion and video streams will be acquired from surgeons during simulated surgical tasks using the da Vinci surgical robot. Surgical tasks will be represented using grammars on a set of simple, elementary surgical gestures modeled using finite-state machines. Both the elementary gestures and the task-dependent grammatical constraints will be inferred directly from data. A key element of the proposed approach is to first develop accurate models from simple, low-dimensional datasets such as direct measurements of surgical motion, and then to use these models to boostrap model training to more complex, high-dimensional synchronized video of the same procedure.The core research problems to be addressed are thus: Motion sequence modeling for elementary tasks: Develop algorithms that are able to learn finite-state machines modeling simple, context-independent elementary motion or manipulation tasks in terms of atomic gestures using a limited set of manually labeled motion data. Multi-modal models of elementary tasks: Create effective methods for augmenting elementary task models with complementary and contextual features from synchronized video data. Grammar induction: Use elementary task models to automatically label more, larger data sequences and infer high-level grammatical structures from these labelings. Model bootstrapping: Use elementary task and grammatical models to label even larger data corpi that are then used to retrain elemetary task models using only video data.In summary, this project will study the problem of modeling structured human manipulative actions such as those that occur during minimally invasive surgery. Using statistical modeling methods originally developed for speech understanding, models for the language of surgical motion will be created and evaluated. Thesemodels will first be created using motion measurements from a surgical robot, but will then be transferred to video data from the same system. These specific models will have applicability to surgical training, skill assessment, and robotic surgical augmentation. More generally, this project will advance our ability to develop structured models for understanding com-plex data sources, which has wide ranging implications. It will facilitate a variety of applications that require understanding of human action such gesture-based systems for human-machine interaction, methods for interpreting or detecting action in video surveillance data, or assessment and measurement of movement for bio-mechanics and medicine. At the same time, it will advance the basic science of modeling large and complex data streams such as video, text, speech, or medical images.URL: http://www.cs.jhu.edu/CIRL/projects/SurgicalModeling/