Neurally and ocularly informed graph-based models for searching 3D environments
Neurally and ocularly informed graph-based models for searching 3D environments
复制标题
DOI：
10.1088/1741-2560/11/4/046003
复制
发表时间：
2014-08-01
影响因子：
4
通讯作者：
Sajda, Paul
中科院分区：
文献类型：
--
作者：
Jangraw, David C.;Wang, Jun;Sajda, Paul
Objective.当我们在一个环境中移动时，我们不断地对我们遇到的事情进行评估，判断和决定。有些是立即采取行动，但更多的是成为精神笔记或短暂的印象-我们对世界的隐含'标签'。在本文中，我们使用这种标记的生理相关性来构建一个混合脑机接口（hBCI）系统，用于3D环境的有效导航。Approach.首先，我们记录脑电图（EEG），扫视和瞳孔数据从受试者，因为他们通过一个小部分的自由观看条件下的3D虚拟城市移动。使用机器学习，我们整合了他们遇到的物体所引起的神经和视觉信号，以推断哪些是他们主观感兴趣的。这些推断出的标签通过城市中对象的大型计算机视觉图传播，使用半监督学习来识别与标记的对象在视觉上相似的其他看不见的对象。最后，系统绘制一条有效的路线，帮助受试者访问它所识别的“相似”对象。主要结果。我们表明，通过利用受试者的隐式标记来寻找感兴趣的对象，而不是天真地探索，中值搜索精度从25%提高到97%，并且中值受试者只需要旅行40%的距离就可以看到84%的感兴趣的对象。我们还发现，神经信号和视觉信号以互补的方式对分类器对被试内隐标记的推断做出贡献。意义总之，我们表明，反映3D环境中对象的主观评估的神经和视觉信号可用于通知该环境的基于图形的学习模型，从而产生一个hBCI系统，该系统可以改善特定于用户兴趣的导航和信息传递。
Objective. As we move through an environment, we are constantly making assessments, judgments and decisions about the things we encounter. Some are acted upon immediately, but many more become mental notes or fleeting impressions-our implicit 'labeling' of the world. In this paper, we use physiological correlates of this labeling to construct a hybrid brain-computer interface (hBCI) system for efficient navigation of a 3D environment. Approach. First, we record electroencephalographic (EEG), saccadic and pupillary data from subjects as they move through a small part of a 3D virtual city under free-viewing conditions. Using machine learning, we integrate the neural and ocular signals evoked by the objects they encounter to infer which ones are of subjective interest to them. These inferred labels are propagated through a large computer vision graph of objects in the city, using semi-supervised learning to identify other, unseen objects that are visually similar to the labeled ones. Finally, the system plots an efficient route to help the subjects visit the 'similar' objects it identifies. Main results. We show that by exploiting the subjects' implicit labeling to find objects of interest instead of exploring naively, the median search precision is increased from 25% to 97%, and the median subject need only travel 40% of the distance to see 84% of the objects of interest. We also find that the neural and ocular signals contribute in a complementary fashion to the classifiers' inference of subjects' implicit labeling. Significance. In summary, we show that neural and ocular signals reflecting subjective assessment of objects in a 3D environment can be used to inform a graph-based learning model of that environment, resulting in an hBCI system that improves navigation and information delivery specific to the user's interests.