Scaling Volumetric Imaging, Analysis and Science Communication Using Immersive Virtual Reality

在过去的15年里，高通量成像的新显微镜技术和方法 通过在3年内扩展数据集的分辨率和规模，彻底改变了结构生物学 尺寸。产生的映像卷通常为数百GB甚至数十TB 而对于大体积的脑部电子显微镜图像，可以接近PB大小。这些文件 大小对图像分析和具有代表性的RAW集合的交流构成了挑战 数据和量化。大文件包含许多结构，需要机器学习(ML) 在允许纠错的上下文中的策略。科学交流需要工具 随时获取原始数据，以更高效的方式进行快速积累的沟通 科学信息的集合。快速积累的数字图书馆也为 教学和培训，这在很大程度上是未开发的。 我们建议利用虚拟现实(VR)来转变这些挑战，充分利用 关于立体视觉和模式识别的自然能力，以及对科学的 交流、教学和培训、听觉处理以处理语言和本地化 听起来。基于工具库和我们拥有的大型文件的直接体积渲染 建立在我们的虚拟现实软件SyGlass中，我们将首先扩展现代领域的学习和 ML领域中的所谓元学习技术，用于分析迭代次数较少的图像 对象计数到对象跟踪和跟踪(目标1)。接下来，我们将利用新的 用于云渲染的技术可显著降低采用 赛格拉斯(目标2)。最后，我们将提供新的工具来有效地生成叙述性科学 用于实验室环境、手稿出版物和生产的VR演示文稿 教材(目标3)。大脑的复杂性为以下方面提供了一个具有挑战性的试验台 教学和培训。在这些目标中的每一个，我们都将在分析 大数据量，以及与同事和非专家交流3D和4D数据。

Over the past 15 years, new microscope technologies and methods for high throughput imaging have revolutionized structural biology by extending the resolution and scale of datasets in 3 dimensions. The resulting image volumes are more typically hundreds of GB to even tens of TB and for large volume electron microscope images of brain, can approach PB sizes. These file sizes pose challenges for image analysis, and communication of a representative set of raw data and quantification. Large files contain many structures, and require machine learning (ML) strategies in a context that permits error correction. Scientific communication requires tools for ready access to raw data, and more efficient methods to communicate the rapidly accumulating sets of scientific information. The rapidly accumulating digital library also affords a resource for teaching and training, which is largely untapped. We propose to leverage virtual reality (VR) to transform each of these challenges, capitalizing on natural abilities for stereoscopic vision and pattern recognition and, for scientific communication, teaching and training, auditory processing to process language and localize sounds. Based upon the tool base and direct volume rendering of large files that we have established in our VR software, called syGlass, we will first expand modern domain learning and so-called meta-learning techniques in the ML field to analyze images with few iterations from object counting to object tracking and tracing (Aim 1). Next, we will capitalize on new technologies for cloud rendering to significantly mitigate the hardware costs for adoption of syGlass (Aim 2). Finally, we will provide novel tools to efficiently generate narrated scientific presentations in VR for use in the lab setting, as manuscript publications, and for production of educational materials (Aim 3). The complexity of the brain offers a challenging testbed for teaching and training. In each of these Aims, we will introduce paradigm shifts in the analysis of the large data volumes, and communication of 3D and 4D data to colleagues and non-experts.