Integrating Human and Machine Learning for Enabling Co-Adaptive Body-Machine Interfaces

集成人类和机器学习以实现自适应体机接口

• 批准号: 2054406
• 负责人: Ferdinando Mussa-Ivaldi
• 金额: $ 71.42万
• 依托单位: Rehabilitation Institute of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2054406&HistoricalAwards=false
• 关键词: Integrating; Human; Machine; Learning; Enabling

This project for the Mind, Machine, and Motor Nexus (M3X) program will advance understanding of how people learn new neuromotor skills, and subsequently apply that understanding to the creation of innovative wearable device controllers called body-machine interfaces (BoMIs). Individuals with neuromuscular impairment -- perhaps due to stroke or spinal cord injury -- may have difficulty carrying out the activities of everyday life. This project explores novel interfaces through which an individual can use their body's residual mobility to issue commands to assistive devices such as computer cursors, wheelchairs, or robotic arms. The project has three main research goals. The first is to improve existing methods for translating small body movements into controller commands for assistive devices. The second is to model the process by which the human user learns over time to use the body-machine interface. The third is to apply the obtained model of the learning process to enable the body-machine interface to adjust to the evolving characteristics of the human user. An interface that does not adapt to changes in its user may significantly degrade in performance over time. On the other hand, an interface whose properties instantly change with every small shift in user behavior will be difficult to control. The ultimate outcome of this project will be human-machine interfaces based on body movement that consider the user and the interface as two components of an integrated system in which each component continually learns from and adapts to the other. The results of the project will lead to assistive devices that more affordable, and provide more versatile control and ease of use. The underlying principles of co-adaptation to be identified through this work are also relevant to rehabilitation from disease or injury, as well as to increasing the capabilities of human-operated robotic systems.Recent work has shown that linear methods such as principal component analysis (PCA) may be effectively used in a body-machine interface (BoMI) to map elements from a higher dimensional feature space of body movements onto a lower dimensional space of device commands. In this project, the features that provide input to the BoMI are generated by multiple inertial measurement units (IMUs) worn by the user; the IMUs report their current orientation in an inertial reference space. The output from the BoMI are commands used to control a sequence of representative devices, specifically a computer cursor, a simulated wheelchair, an actual wheelchair, and a simulated manipulator arm. The three technical goals of the project are as follows: 1) Compare the performance of a linear map based on PCA to a nonlinear map based on an autoencoder network (AEN) for providing input features to the BoMI that translates residual mobility space features into device commands. The AEN is capable of representing a richer variety of features than PCA, but it remains to be shown, for example, whether human users can make effective use of that variety. 2) Obtain a computable representation of the process by which humans learn neuromotor skills. This representation will be based on the premise that humans simultaneously learn both a forward and inverse map of the relationship between neuromotor signals and the resulting physical outcomes. Once learned, the forward map predicts the outcomes that will result from a certain set of signals, while the inverse map is used to generate the signals that correspond to a given desired physical outcome. As a person learns mastery of a neuromotor skill, the forward and inverse maps become more accurate predictors of actual behavior, and the degree of learning can be monitored through estimates of these maps. 3) Incorporate a co-adaptation algorithm into the BoMI for maintaining performance as the user's mastery of the BoMI changes. In most current approaches to human-machine interfaces, the interface is fixed following an initial calibration stage, and the user must learn to control that interface configuration. In this project, the learning representation of objective (2) will be used to monitor and periodically update the BoMI map parameters. The implementation of this objective is aided by parallels between the human learning model and the AEN training method, which automatically generates a decoder network that captures the inverse map between desired device commands and the corresponding residual mobility features needed to produce them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个项目的思想，机器和运动的关系（M3 X）计划将推进人们如何学习新的神经运动技能的理解，并随后将这种理解应用到创建创新的可穿戴设备控制器称为身体-机器接口（BoMI）。患有神经肌肉损伤的人-可能是由于中风或脊髓损伤-可能难以进行日常生活活动。这个项目探索了新的界面，通过它，个人可以使用他们身体的剩余移动性来向辅助设备（如计算机光标，轮椅或机器人手臂）发出命令。该项目有三个主要研究目标。第一个是改进现有的方法，将微小的身体运动转化为辅助设备的控制器命令。第二个是对人类用户随着时间的推移学习使用身体-机器界面的过程进行建模。第三是应用所获得的学习过程的模型，以使身体-机器接口能够适应人类用户的演变特征。不能适应用户变化的界面可能会随着时间的推移而显著降低性能。另一方面，一个界面的属性会随着用户行为的每一个小的变化而立即改变，这将是难以控制的。该项目的最终成果将是基于身体运动的人机界面，该界面将用户和界面视为集成系统的两个组件，其中每个组件不断学习并适应其他组件。该项目的结果将导致辅助设备，更负担得起的，并提供更多功能的控制和易用性。通过这项工作确定的共同适应的基本原则也与疾病或伤害的康复有关，最近的工作表明，线性方法，如主成分分析（PCA），可以有效地用于人体-机器接口（BoMI）以将元素从身体运动的较高维度特征空间映射到设备命令的较低维度空间。在该项目中，向BoMI提供输入的特征由用户佩戴的多个惯性测量单元（伊穆斯）生成;伊穆斯单元报告其在惯性参考空间中的当前方位。BoMI的输出是用于控制一系列代表性设备的命令，特别是计算机光标，模拟轮椅，实际轮椅和模拟机械臂。该项目的三个技术目标如下：1）比较基于PCA的线性映射与基于自动编码器网络（AEN）的非线性映射的性能用于向BoMI提供输入特征，BoMI将剩余移动性空间特征转换为设备命令。AEN能够表示比PCA更丰富的各种特征，但仍有待证明，例如，人类用户是否可以有效地利用这种多样性。2)获得人类学习神经运动技能过程的可计算表示。这种表示将基于这样的前提，即人类同时学习神经运动信号和由此产生的身体结果之间关系的正向和反向映射。一旦学习，前向映射预测将由某组信号产生的结果，而反向映射用于生成对应于给定的期望物理结果的信号。当一个人学会掌握神经运动技能时，正向和反向映射会成为实际行为的更准确的预测因子，并且可以通过对这些映射的估计来监测学习程度。3)在BoMI中加入一个自适应算法，以便在用户掌握BoMI的情况发生变化时保持性能。在大多数当前的人机界面方法中，界面在初始校准阶段之后是固定的，并且用户必须学会控制该界面配置。在这个项目中，目标（2）的学习表示将用于监控和定期更新BoMI地图参数。这一目标的实现得益于人类学习模型和AEN训练方法之间的相似之处，该奖项反映了NSF的法定使命，并通过利用基金会的智力价值和更广泛的影响力进行评估，被认为值得支持审查标准。