CDS&E: Immersive Virtual Reality for Discovering Hidden Chemical Information and Improving Multivariate Modeling and Predication

在化学学部化学测量与成像（CMI）项目的支持下，爱达荷州立大学John Kalivas教授和他的本科团队正在开发新的化学数据分析的虚拟现实（VR）方法，以及来自刺激竞争研究的既定项目（EPSCoR）和以人为本的计算项目的部分共同资助。与爱达荷国家实验室的应用可视化实验室合作，Kalivas团队正在用触觉和听觉元素补充数据可视化VR过程，以增强数据分析。通过将人类的三种感官与人类独特的推理能力结合起来，在计算机仅限于数字结论的复杂数据分析情况下，它应该有可能超越计算机。在许多数据分析情况下，必须做出关键决策，但这些决策并不总是显而易见的，例如决定早期活检是否为癌症。除了医疗诊断之外，所获得的见解还将用于过程分析技术和农业等广泛的其他应用。VR项目的教育版本正在开发中，让盲人和聋哑学生学习和探索数据科学过程，将这些学生带到机器学习技术的最前沿。这部分工作是与爱达荷州聋哑人和盲人教育服务机构合作开发的。该项目将为爱达荷州服务不足地区的本科生和高中生提供研究机会。在这个项目中，Kalivas团队正在利用他们最近在VR中开发的物理化学反应集成光谱测量（PRISM）方法。PRISM的一个战略特征是，它由数百个样本相似性测量组成，这些相似性测量基于放大在测量光谱（如红外（IR），近红外，拉曼等）中编码的隐藏但必要的化学（和物理化学，如果存在）属性。这些相似性测量中的每一个都可以作为VR设置中的对象特征合并，以充分表征样品的固有化学性质。该探索项目的一个关键目标是有效地将多维化学数据转换为虚拟现实，从而显著增强数据与发现之间的联系。使用VR在视觉上、触觉上（触摸包括纹理到数据形状）和声音（每个数据点的声音）描绘数据点/样本，人类用户将能够深入探索数据，发现隐藏的模式和相应的化学属性。有了这样一个详细的数据分析过程，应该可以通过更好的解释或解释样本关系来改进分类和目标样本预测。所有开发的算法都将发布在Kalivas网站上，允许潜在用户免费访问。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

With support from the Chemical Measurement and Imaging (CMI) program in the Chemistry Division, and partial co-funding from the Established Program to Stimulate Competitive Research (EPSCoR) and the Human-Centered Computing Program, Professor John Kalivas and his undergraduate group at Idaho State University are developing new virtual reality (VR) methods of chemical data analysis. In collaboration with the Applied Visualization Laboratory at the Idaho National Laboratory, the Kalivas group is complementing data visualization VR processes with tactile and auditory elements to enhance data analysis. By including all three human senses in conjunction with the human’s unique reasoning capabilities, it should be possible to outperform the computer in complex data analysis situations where the computer is restricted to numerical conclusions. There are many data analysis situations where critical decisions must be made that are not always obvious, such as deciding if an early biopsy is cancerous or not. In addition to medical diagnostics, the insights gained will be useful for a broad range of other applications such as process analytical technologies and the agriculture industry. Educational versions of the VR programs are being developed to allow blind and deaf students to learn and explore data science processes, bringing these students to the forefront of machine learning technology. This part of the work is being developed in collaboration with the Idaho Educational Services for the Deaf and Blind. The project will provide research opportunities for undergraduates and high school students in underserved regions of Idaho.In this project, the Kalivas team is exploiting their recently developed Physicochemical Response Integrated Spectral Measurement (PRISM) approach in VR. A strategic feature of PRISM is that it is composed of hundreds of sample-wise similarity measures based on amplifying hidden-but-essential chemical (and physiochemical if present) properties encoded within measured spectra such as infrared (IR), near IR, Raman, etc. Each of these similarity measures can be incorporated as an object feature in the VR setting to fully characterize the inherent chemical nature of a sample. A key goal of this exploratory project is to effectively convert multidimensional chemical data to VR thereby significantly enhancing the connection between data and discovery. Using VR to depict data points/samples visually, haptically (touching including texture to data shapes), and with sonification (sound for each data point), the human user will be able to explore data in-depth to uncover hidden patterns and corresponding chemical attributes. With such a detailed data analysis process, improved classification and target sample predictions with better explanations or interpretations of sample relationships should be possible. All developed algorithms will be posted to the Kalivas web site, allowing free access to potential users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.