Streamlining Volumetric Imaging, Analysis and Publication Using Immersive Virtual Reality

在过去的15年里，用于高通量成像的新成像技术和方法 通过扩大所收集图像的分辨率和规模， 在三维空间里。生成的映像卷通常为数百GB甚至数十GB， TB，在某些情况下接近PB大小。这些文件大小对映像 代表性原始数据集的采集、图像分析和通信，以及 量化图像采集运行可能是漫长和昂贵的，并且通常错误不是 直到扫描完成后才能识别。大型文件包含许多结构，并且需要 机器学习（ML）策略在允许纠错的上下文中。科学 通信需要随时访问原始数据的工具，以及更有效的方法， 交流快速积累的科学信息。我们建议利用 虚拟现实（VR）和虚拟现实环境中的口头交流，以简化每个 科学工作的这些阶段，通过利用更自然的立体能力， 视觉和听觉来处理场景和语言。基于工具库和直接 我们已经在我们的VR软件中建立了大文件的体积渲染，称为syGlass，我们 将首先将VR集成到显微镜控制中，以调整显微镜，然后有效地 在获取图像时检查3D图像（目标1）。接下来，我们将介绍新领域 ML领域的自适应技术，以扩大当前3D图像量化能力 通过将其与不需要ML的用户优化体验相结合， 专业知识，但提供自动化和准确的结果（目标2）。最后，我们将提供工具 在VR中有效地生成用于实验室环境的叙述式科学演示， 手稿出版物和制作教育材料（目标3）。在这些 活动，我们将介绍实验管理的范式转变，分析 结果数据，并向其他科学家和一般出版手稿和材料， 公众

Over the past 15 years, new imaging technologies and methods for high throughput imaging have revolutionized structural biology by extending the resolution and scale of collected images in 3 dimensions. The resulting image volumes are more typically hundreds of GB to even tens of TB and in some cases approach PB sizes. These file sizes pose challenges for image acquisition, image analysis, and communication of a representative set of raw data and quantification. Image acquisition runs can be lengthy and expensive, and often errors are not identified until after the completion of scanning. Large files contain many structures, and require machine learning (ML) strategies in a context that permits error correction. Scientific communication requires tools for ready access to raw data, and more efficient methods to communicate the rapidly accumulating sets of scientific information. We propose to leverage virtual reality (VR) and verbal communication within the VR environment, to streamline each of these stages of scientific work, by capitalizing on the more natural abilities for stereoscopic vision and hearing to process scenes and language. Based upon the tool base and direct volume rendering of large files that we have established in our VR software, called syGlass, we will first integrate VR into the microscope controls for tuning the microscope and then efficiently inspecting images in 3D as they are acquired (Aim 1). Next, we will introduce novel domain adaptation techniques in the ML field to scale up 3D image quantification capabilities for current acquisition sizes, by coupling them with user-optimized experiences that do not require ML expertise, and yet provide automated and accurate results (Aim 2). Finally, we will provide tools to efficiently generate narrated scientific presentations in VR for use in the lab setting, as manuscript publications, and for production of educational materials (Aim 3). In each of these activities, we will introduce paradigm shifts in the management of experiments, analysis of the resulting data, and publication of manuscripts and materials to other scientists and the general public.