Multimodal Interaction with Biological Molecules in Virtual Environments

高级计算科学的一个核心问题是使科学家和工程师能够理解大型数据集和大型模拟的结果。传达这种理解的最著名的方法是可视化，它允许用户看到数据。然而，某些数据的复杂性和性质使得它适合使用其他感官来表示。例如，形状可以通过触摸来确定。本项目研究大规模复杂科学计算问题的多感官虚拟环境。为了更好地集中研究工作和验证结果，它使用多模态交互与复杂的大分子数据集作为驱动应用。它旨在设计技术和工具，以便在具有视觉，听觉和触觉（力）反馈的虚拟环境中与非常大的生物分子数据集进行有意义的交互。大多数生物分子数据集由几个基本元素（如原子类型、氨基酸侧链和核酸碱基）组成，这些基本元素组合在一起产生生物复杂性的丰富嵌合体。研究人员认为，这足够丰富，可以代表其他科学可视化分支学科，但也足够专业，可以提供一个明确的研究目标。该研究包括在顺序和并行架构上使用感知相关的多分辨率渲染的多模态反馈。提出的进展优化了多模态渲染过程，使用了一组在大多数生物分子数据集中常见的三个特征——有限的基序复杂性、有限的精度和可变的可感知细节水平。大多数科学可视化数据集至少具有上述两个特征，因此，在此方面的进展应进一步推动虚拟现实辅助科学计算的其他领域的前沿研究。在这里，研究人员将使用普通主题的分层预渲染表示，以在多模态环境中实现交互速度的真实感。他们还将利用处理器对可变分辨率整数算法的支持，以及生物分子数据的有限动态范围和有限输入精度，通过以计算精度换取速度，透明地加快多模态渲染。最后，他们将利用人们的感官对最不易察觉的刺激有阈值这一事实，并且他们通常只在有限的区域内感知高细节，探索在跨多种感官的集成和连贯框架中解决可变细节渲染的技术。研究人员将通过与两名从事蛋白质折叠和合理药物设计的校园生物化学研究人员以及其他访问科学家和学者合作，在现实数据集上验证所提出方法的有效性。

A central problem of advanced computational science is making large datasets and the results of large simulations understandable by scientists andengineers. The best-known means of imparting this understanding isvisualization, which allows users to see the data. However, the complexityand nature of some data makes it appropriate to represent using the othersenses. For example, shape can be determined by touch. This project studiesmultisensory virtual environments for large--scale complex scientificcomputing problems. To better focus the research efforts and validate theresults, it uses multimodal interaction with complex big-molecular datasetsas the driving application. It aims to devise techniques and tools formeaningful interactions with very large biomolecular data sets in virtualenvironments with visual, aural, and haptic (force) feedback. Mostbiomolecular datasets consist of a few primitives (such as atom types, ammoacid sidechains, and nucleic acid bases) that are combined to yield richmosaics of biological complexity. The researchers believe this is richenough to be representative of other scientific visualizationsub-disciplines and yet specialized enough to offer a well-defined researchtarget.The research includes multimodal feedback using perception-dependentmultiresolution rendering on sequential and parallel architectures. Theproposed advances optimize the multimodal rendering processes using a setof three characteristics common across most biomolecular datasets - limitedmotif complexity, bounded accuracy, and variable level of perceptibledetail. Most scientific visualization datasets share at least two of thesecharacteristics and therefore advances here should further the cutting-edgeresearch in other areas of virtual-reality-assisted scientific computing.Here, the researchers will use hierarchical pre-rendered representations ofcommon motifs to allow realism at interactive speeds in multimodalcontexts. They will also take advantage of processor support for variableresolution integer arithmetic and the limited dynamic range and boundedinput accuracy of biomolecular data to transparently speed up multimodalrendering by trading computational accuracy for speed. Finally, they willtake advantage of the fact that people's senses have thresholds of theleast perceptible stimuli, and that they typically perceive high details inonly a limited region, to explore techniques to address variablelevel-of-detail renderings in an integrated and coherent framework acrossmultiple senses. The researchers will validate the efficacy of proposedapproaches on real-life datasets through a collaboration with two on-campusbiochemistry researchers working on protein folding and rational drugdesign and other visiting scientists and scholars.