OAC Core: Small: Shape-Image-Text: A Data-Driven Joint Embedding Framework for Representing and Analyzing Large-Scale Brain Microvascular Data

如今，科学和工程的许多领域在从不断增长的可用数据中综合信息方面面临着越来越多的挑战。当数据类型和格式变化时，如三维对象的情况，这些挑战甚至更加复杂。这些对象可以通过不同的表示（模态）来描述：形状，图像和文本。最近，深度学习方法被证明是一种有效的处理技术，可以在不依赖硬编码指标的情况下暴露对象关系。然而，这些方法侧重于单一模式。该项目旨在设计和开发一种模型，将三维形状，图像和文本等多种类型的数据统一到一个定量模型中。遵循严格的方法，来自韦恩州立大学的研究人员团队将把多模态和异构的表示和特征映射到一个通用的高维编码空间，其特征是统一的表示和度量。然后，该团队将通过将研究结果应用于与地区健康科学专业人员合作收集的微型磁共振成像（MICRO-MRI）微血管数据来验证这项工作。该项目弥合了神经科学数据分析的重大差距，并将产生一个网络基础设施框架，以刺激该领域的研究。该项目还将为本科生和研究生提供教育活动，并推广到当地中学生。正如NSF的使命所述，该项目服务于国家利益：促进科学进步;该提案的研究目标围绕统一理论的多模态数据驱动的联合嵌入框架，涉及高维多模态特征向量、基于概率的联合嵌入和深度神经网络的设计，从而使得从一个全新的角度有效地表示和处理大规模微血管网络成为可能。所提出的深度神经网络的计算实现可以将从大型数据集获得的具有异构成像，文本和其他特征的三维形状转换为新的高维等距多视图（形状，图像和文本）概率空间。建议的联合嵌入空间保留了所有固有的几何，成像和文本特征，并有能力整合其他多模态属性。通过统一度量向量场的广义联合嵌入空间允许对3D多模态数据信息学中密集涉及的形状处理和测量中的几何可扩展性和可变性进行正式和多样的研究。在所提出的联合嵌入空间中，通过使用统一度量，可以容易地计算和测量全局和局部形状比较和分析，这将显著提高系统的自动化程度，减少人类的干预，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

Today, many areas of science and engineering face increased challenges in synthesizing information from the ever-growing amount of data available. Such challenges are even more complex when the data type and format vary, as is the case for three-dimensional objects. Such objects can be described by different representations (modalities): shapes, images, and texts. Recently, deep learning methods were shown to be effective processing techniques by exposing object relationships without relying on hard-coded metrics. However, such methods focus on single modalities. This project seeks to design and develop a model that unifies multiple types of data such as three-dimensional shapes, images and text in a single quantitative model. Following a rigorous approach, the team of researchers from Wayne State University will map the multimodal and heterogeneous representations and features onto a universal high-dimensional encoding space, characterized by uniform representation and metric. The team will then validate the work by applying the research results to MICRO Magnetic Resonance Imaging (MICRO-MRI) microvascular data collected in collaboration with area health science professionals. The project bridges a significant gap in neuroscience data analysis and will produce a cyberinfrastructure framework that will stimulate research in the field. The project will also provide educational activities for undergraduate and graduate students, as well as outreach to local middle school students. This project serves the national interest, as stated by NSF's mission: to promote the progress of science; to advance the national health, prosperity and welfare.The research goal of this proposal centers around the unified theoretical multimodality data-driven joint embedding framework and involves design of a high-dimensional multimodal feature vector, probability-based joint embedding, and deep neural networks, hence making it possible to effectively represent and process the large-scale microvascular networks from a brand-new perspective. The proposed computational realization of deep neural networks can transform a three-dimensional shape with heterogeneous imaging, textual, and other features obtained from a large dataset to a novel high-dimensional isometric multi-view (shape, image, and text) probability space. The proposed joint embedding space preserves all intrinsic geometric, imaging, and textual characteristics and has the capability to integrate other multimodality properties. The generalized joint embedding space through the unified metric vector field allows formal and diverse study of geometry scalability and variability in shape processing and measurement intensively involved in 3D multimodal data informatics. In the proposed joint embedding space, the global and local shape comparison and analysis can be easily computed and measured by using the unified metric, which will significantly increase system's automation, reduce human's interventions, and discover new knowledge in vascular diseases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.