Integrating Human and Machine Learning for Enabling Co-Adaptive Body-Machine Interfaces

心智、机器和运动Nexus （M3X）项目将推进对人们如何学习新的神经运动技能的理解，并随后将这种理解应用于创新的可穿戴设备控制器（称为身体-机器接口（BoMIs））的创建。患有神经肌肉损伤的人——可能是由于中风或脊髓损伤——可能难以进行日常生活活动。这个项目探索了一种新颖的接口，通过这种接口，个人可以利用他们身体的剩余活动能力向辅助设备（如电脑光标、轮椅或机械臂）发出命令。该项目有三个主要的研究目标。首先是改进现有的方法，将微小的身体动作转化为辅助设备的控制器命令。第二种是模拟人类用户随着时间的推移学习使用身体-机器界面的过程。第三是应用获得的学习过程模型，使人体-机器界面能够适应人类用户不断变化的特征。随着时间的推移，不能适应用户变化的界面可能会显著降低性能。另一方面，如果一个界面的属性会随着用户行为的微小变化而瞬间改变，那么它就很难控制。该项目的最终成果将是基于身体运动的人机界面，将用户和界面视为集成系统的两个组成部分，其中每个组成部分不断学习并适应另一个。该项目的结果将导致辅助设备更实惠，并提供更多功能的控制和易用性。通过这项工作确定的共同适应的基本原则也与疾病或损伤的康复以及提高人类操作的机器人系统的能力有关。最近的研究表明，主成分分析（PCA）等线性方法可以有效地用于身体-机器界面（BoMI），将身体运动的高维特征空间中的元素映射到设备命令的低维空间。在本项目中，为BoMI提供输入的特征由用户佩戴的多个惯性测量单元（imu）产生；imu在惯性参考空间中报告其当前方向。BoMI的输出是用于控制一系列代表性设备的命令，特别是计算机光标、模拟轮椅、实际轮椅和模拟机械臂。该项目的三个技术目标如下：1)比较基于PCA的线性地图与基于自动编码器网络（AEN）的非线性地图的性能，后者为BoMI提供输入特征，将剩余移动空间特征转换为设备命令。与PCA相比，AEN能够表示更丰富的特征，但它仍有待证明，例如，人类用户是否能够有效地利用这种多样性。2)获得人类学习神经运动技能过程的可计算表示。这种表示将基于这样一个前提，即人类同时学习神经运动信号与由此产生的物理结果之间关系的正反映射。一旦学会了，正向映射预测了一组特定信号将产生的结果，而反向映射用于生成与给定的期望物理结果相对应的信号。当一个人掌握一项神经运动技能时，正向和反向映射就能更准确地预测实际行为，并且可以通过对这些映射的估计来监测学习的程度。3)在BoMI中加入一种自适应算法，以便在用户对BoMI的掌握发生变化时保持性能。在大多数当前的人机界面方法中，界面在初始校准阶段之后是固定的，用户必须学会控制该界面配置。在本项目中，将使用目标(2)的学习表示来监测并定期更新BoMI地图参数。人类学习模型和AEN训练方法之间的相似之处有助于实现这一目标，AEN训练方法自动生成一个解码网络，该网络捕获所需设备命令和产生它们所需的相应剩余移动特征之间的逆映射。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project for the Mind, Machine, and Motor Nexus (M3X) program will advance understanding of how people learn new neuromotor skills, and subsequently apply that understanding to the creation of innovative wearable device controllers called body-machine interfaces (BoMIs). Individuals with neuromuscular impairment -- perhaps due to stroke or spinal cord injury -- may have difficulty carrying out the activities of everyday life. This project explores novel interfaces through which an individual can use their body's residual mobility to issue commands to assistive devices such as computer cursors, wheelchairs, or robotic arms. The project has three main research goals. The first is to improve existing methods for translating small body movements into controller commands for assistive devices. The second is to model the process by which the human user learns over time to use the body-machine interface. The third is to apply the obtained model of the learning process to enable the body-machine interface to adjust to the evolving characteristics of the human user. An interface that does not adapt to changes in its user may significantly degrade in performance over time. On the other hand, an interface whose properties instantly change with every small shift in user behavior will be difficult to control. The ultimate outcome of this project will be human-machine interfaces based on body movement that consider the user and the interface as two components of an integrated system in which each component continually learns from and adapts to the other. The results of the project will lead to assistive devices that more affordable, and provide more versatile control and ease of use. The underlying principles of co-adaptation to be identified through this work are also relevant to rehabilitation from disease or injury, as well as to increasing the capabilities of human-operated robotic systems.Recent work has shown that linear methods such as principal component analysis (PCA) may be effectively used in a body-machine interface (BoMI) to map elements from a higher dimensional feature space of body movements onto a lower dimensional space of device commands. In this project, the features that provide input to the BoMI are generated by multiple inertial measurement units (IMUs) worn by the user; the IMUs report their current orientation in an inertial reference space. The output from the BoMI are commands used to control a sequence of representative devices, specifically a computer cursor, a simulated wheelchair, an actual wheelchair, and a simulated manipulator arm. The three technical goals of the project are as follows: 1) Compare the performance of a linear map based on PCA to a nonlinear map based on an autoencoder network (AEN) for providing input features to the BoMI that translates residual mobility space features into device commands. The AEN is capable of representing a richer variety of features than PCA, but it remains to be shown, for example, whether human users can make effective use of that variety. 2) Obtain a computable representation of the process by which humans learn neuromotor skills. This representation will be based on the premise that humans simultaneously learn both a forward and inverse map of the relationship between neuromotor signals and the resulting physical outcomes. Once learned, the forward map predicts the outcomes that will result from a certain set of signals, while the inverse map is used to generate the signals that correspond to a given desired physical outcome. As a person learns mastery of a neuromotor skill, the forward and inverse maps become more accurate predictors of actual behavior, and the degree of learning can be monitored through estimates of these maps. 3) Incorporate a co-adaptation algorithm into the BoMI for maintaining performance as the user's mastery of the BoMI changes. In most current approaches to human-machine interfaces, the interface is fixed following an initial calibration stage, and the user must learn to control that interface configuration. In this project, the learning representation of objective (2) will be used to monitor and periodically update the BoMI map parameters. The implementation of this objective is aided by parallels between the human learning model and the AEN training method, which automatically generates a decoder network that captures the inverse map between desired device commands and the corresponding residual mobility features needed to produce them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.