OAC Core: Small: Shape-Image-Text: A Data-Driven Joint Embedding Framework for Representing and Analyzing Large-Scale Brain Microvascular Data

今天，许多科学和工程领域在从不断增长的可用数据中合成信息方面面临着越来越大的挑战。当数据类型和格式变化时，这样的挑战甚至更加复杂，就像三维对象的情况一样。这样的对象可以通过不同的表示(形态)来描述：形状、图像和文本。最近，深度学习方法被证明是一种有效的处理技术，它通过暴露对象关系而不依赖于硬编码的度量。然而，这些方法侧重于单一的模式。该项目旨在设计和开发一种模型，将多种类型的数据统一在一个量化模型中，例如三维形状、图像和文本。按照严格的方法，韦恩州立大学的研究团队将把多模式和不同种类的表示和特征映射到一个通用的高维编码空间，其特征是统一的表示和度量。然后，该团队将通过将研究结果应用于与地区卫生科学专业人员合作收集的微磁共振成像(Micro-MRI)微血管数据来验证工作。该项目弥合了神经科学数据分析方面的重大差距，并将产生一个将刺激该领域研究的网络基础设施框架。该项目还将为本科生和研究生提供教育活动，并向当地中学生提供外联服务。该项目服务于国家利益，正如国家科学基金会的使命所表明的：促进科学进步；促进国家健康、繁荣和福祉。该建议的研究目标围绕统一的理论多模数据驱动的联合嵌入框架，涉及设计高维多模特征向量、基于概率的联合嵌入和深度神经网络，从而从全新的角度有效地表示和处理大规模微血管网络。所提出的深度神经网络的计算实现可以将从大数据集中获得的具有不同成像、文本等特征的三维形状转换到新的高维等距多视(形状、图像和文本)概率空间。所提出的联合嵌入空间保留了所有固有的几何、图像和文本特征，并具有集成其他多模特性的能力。通过统一的度量向量场的广义联合嵌入空间允许形式化和多样化地研究形状处理和测量中的几何、可伸缩性和可变性，深入地涉及3D多模式数据信息学。在建议的联合嵌入空间中，使用统一的度量可以轻松地计算和测量全局和局部形状的比较和分析，这将显著提高系统的自动化程度，减少人为干预，并发现血管疾病的新知识。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today, many areas of science and engineering face increased challenges in synthesizing information from the ever-growing amount of data available. Such challenges are even more complex when the data type and format vary, as is the case for three-dimensional objects. Such objects can be described by different representations (modalities): shapes, images, and texts. Recently, deep learning methods were shown to be effective processing techniques by exposing object relationships without relying on hard-coded metrics. However, such methods focus on single modalities. This project seeks to design and develop a model that unifies multiple types of data such as three-dimensional shapes, images and text in a single quantitative model. Following a rigorous approach, the team of researchers from Wayne State University will map the multimodal and heterogeneous representations and features onto a universal high-dimensional encoding space, characterized by uniform representation and metric. The team will then validate the work by applying the research results to MICRO Magnetic Resonance Imaging (MICRO-MRI) microvascular data collected in collaboration with area health science professionals. The project bridges a significant gap in neuroscience data analysis and will produce a cyberinfrastructure framework that will stimulate research in the field. The project will also provide educational activities for undergraduate and graduate students, as well as outreach to local middle school students. This project serves the national interest, as stated by NSF's mission: to promote the progress of science; to advance the national health, prosperity and welfare.The research goal of this proposal centers around the unified theoretical multimodality data-driven joint embedding framework and involves design of a high-dimensional multimodal feature vector, probability-based joint embedding, and deep neural networks, hence making it possible to effectively represent and process the large-scale microvascular networks from a brand-new perspective. The proposed computational realization of deep neural networks can transform a three-dimensional shape with heterogeneous imaging, textual, and other features obtained from a large dataset to a novel high-dimensional isometric multi-view (shape, image, and text) probability space. The proposed joint embedding space preserves all intrinsic geometric, imaging, and textual characteristics and has the capability to integrate other multimodality properties. The generalized joint embedding space through the unified metric vector field allows formal and diverse study of geometry scalability and variability in shape processing and measurement intensively involved in 3D multimodal data informatics. In the proposed joint embedding space, the global and local shape comparison and analysis can be easily computed and measured by using the unified metric, which will significantly increase system's automation, reduce human's interventions, and discover new knowledge in vascular diseases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.