Towards Detecting Pilots' Work Overload From their Brain Signals - Next Generation Cockpit Information System

高压和复杂的信息环境，如战斗机飞行员所经历的那些，对飞行员施加高认知负荷（称为“认知过载”）。这是由于不同的仪器和传感器以各种方式提供了大量潜在的关键信息。飞行员需要关注与可疑事件和目标相关的重要信息，同时仍然保持整体的综合态势感知。在观察、理解和对关键信息做出反应方面的延迟可能意味着生与死的区别。这意味着飞行员必须在正确的时间使用正确的仪器和传感器，并利用这些信息在很短的时间内做出最佳决策。研究表明，当用户同时从事多项任务时，特别是当任务具有不同的性质时，会发生认知过载。例如，飞行员必须与驾驶舱信息系统（视觉模态）进行交互和响应，同时与他们的僚机或命令控制（听觉模态）进行通信。此外，研究表明，用户可能会经历一种模态（例如视觉）的认知过载，同时仍然能够从另一种类型的模态（例如听觉）接收信息并处理它。这一点尤其重要，例如，如果飞行员正在经历视觉认知过载，通过这种模态发送关键任务信息将是无效的，并且很可能被飞行员错过。然而，使用另一种模态（例如听觉）发送相同的信息将允许飞行员在预期时间接收和处理信息。了解飞行员的这些信息将提高驾驶舱信息系统提供关键信息的成功率，从而挽救飞行员的生命。这项研究的总体目标是在确定飞行员的认知过载可以基于一系列大脑激活来检测并通过驾驶舱信息系统对其做出反应方面迈出基础性的第一步。神经心理学的早期研究表明，监测认知负荷是可能的。然而，这项研究仍处于起步阶段。这个项目将通过建立认知超载可以根据一系列大脑激活来检测来推动科学的边界。我们的目标还在于更进一步，从大脑信号中实时检测认知过载的模态，即视觉、听觉或动觉。最后，我们的目标是调查已经确定的大脑激活是否在用户之间是共同的，或者它们对每个人都是独特的。在该项目的过程中，神经科学监测和建模技术将被用来定位当参与者经历认知过载时被激活的大脑区域。然后，人工智能模型将被设计为从（这些局部区域的）大脑信号中学习和检测用户是否正在经历认知过载以及在给定时间认知过载的模式。

High pressured and complex information environments such as those experienced by fighter pilots place high cognitive loads (termed "cognitive overload") on the pilots. This is due to the vast amount of potentially critical information that different instruments and sensors provide in various modalities. Pilots need to focus on important information related to suspicious events and targets, while still maintaining overall comprehensive situational awareness. A delay in observing, comprehending and in turn, reacting to critical information could mean the difference between life and death. This means that pilots have to utilise and engage with the right instrument and sensor at the right time and use that information to make the best possible decision all within a very short space of time. Research has shown that cognitive overload can happen when users are engaged in multiple tasks simultaneously and, in particular, if the tasks have different modality in nature. For example, pilots have to interact and respond to the cockpit information system (visual modality) while they are communicating with their wingmen or command control (auditory modality). Also, research has shown that users can experience cognitive overload for one modality (e.g. visual) while still able to receive information from another type of modality (e.g. auditory) and process it. This is particularly important if, for example, a pilot is experiencing a visual cognitive overload, sending mission-critical information via such modality would be ineffective and highly likely missed by the pilots. However, sending the same information using another modality (e.g. auditory) would allow the pilots to receive and process the information at the intended time. Having this knowledge about pilots will increase the success of delivering critical information by cockpit information systems which could, in turn, save pilots' life. The overall goal of this research is to make a fundamental first step in establishing that pilots' cognitive overload can be detected based on a series of brain activations and reacted upon them by cockpit information systems. Early research in neuropsychology has indicated that it is possible to monitoring cognitive load. However, this research is still in its infancy. This project will push the boundaries of science by establishing that cognitive overload can be detected based on a series of brain activations. We also aim to go one step further and detect the modality of cognitive overload, i.e. either it is visual, auditory or kinaesthetic, in real-time, from the brain signals. Finally, we aim to investigate if the brain activates that have been identified are common across users, or they are unique to each individual. During the course of the project, neuroscience monitoring and modelling techniques will be employed to localise brain regions that get activated when participants are experiencing cognitive overload. Artificial Intelligence models will then be devised to learn and detect from the brain signals (of those localised regions) whether or not a user is experiencing a cognitive overload and what modality of cognitive overload would that be at a given time.