EAGER: Collaborative 3D Materials Science Research in the Cloud

先进材料的发现和广泛应用长期以来一直受到元素和转化路径极其复杂的组合的挑战，这些组合导致了过多的材料特性。 材料研究项目生成的数据集从几十兆字节到兆兆字节不等，因此很难组织、共享和分析。该项目为材料研究带来了新的大数据管理和分析技术，使科学家能够使用新工具访问以前庞大的数据集，通过共享复杂的数据和分析工具来加快创新步伐。该软件将部署在网络上，使世界各地的研究人员能够合作探索材料的结构。 这些工具和方法可以将3维（3-D）和4维（4-D）的实验测量与极端环境下材料的计算建模（如航空航天发动机部件和高效热电材料的开发）相结合。该项目将在多个方向上推动基于网络的大规模图像和数据分析的边界。首先，在异构计算集群上执行复杂的科学工作流将通过利用虚拟化技术和现代集群计算框架（如Apache Spark）来简化。我们将比较实际工作流的不同框架上的并行执行策略的开销。其次，我们将为科学工作流程添加出处跟踪和版本控制，包括基于Web的浏览器，有助于理解过去的分析运行并提高实验的可重复性。我们将为此目的扩展图查询系统和图可视化框架。第三，我们将整合跨参数范围并行运行Dream.3D管道的功能。这将使材料科学界能够快速探索输入参数对分析结果的影响。系统将跟踪子结果，并允许用户浏览和查询元数据（例如，“仪器名称”）和HDF表中的复杂输出数据。将增加一个新的查询系统，涵盖各种形式（表格、图表、文本、图像）。 为了实现这些目标，我们将扩展现有的BisQue图像分析平台，该平台广泛用于大规模图像信息学。BisQue平台和相关的材料研究工具将作为开源发布。

The discovery and widespread implementation of advanced materials have long been challenged by the overwhelmingly complex combinations of elements and transformation paths that result in a plethora of material properties. Materials research projects generate datasets that range from a few tens of Gigabytes to terabytes, making it difficult to organize, share and analyze. This project brings new big data management and analysis techniques to materials research, giving scientists access to previously unwieldy datasets with new tools that accelerate the pace of innovation by allowing sharing of both complex data and analysis tools. The software will be deployed on the web, making it accessible to researchers worldwide to collaboratively explore the structure of materials. These tools and methods allow for the integration of experimental measurements in 3 dimensions (3-D) and 4 dimensions (4-D) with computational modeling of materials in extreme environments, such as aerospace engine components and the development of high efficiency thermoelectric materials.The project will push the boundary of large-scale web-based image and data analysis in multiple directions. First, the execution of complex scientific workflows on heterogeneous compute clusters will be simplified by exploiting virtualization techniques and modern cluster computing frameworks such as Apache Spark. We will compare overheads for parallel execution strategies on different frameworks for realistic workflows. Second, we will add provenance tracking and versioning for scientific workflows including a web-based browser that aids in making sense of past analysis runs and improves repeatability of experiments. We will extend graph query systems and graph visualization frameworks for this purpose. Third, we will integrate the capability to run Dream.3D pipelines in parallel across parameter ranges. This will enable the Materials Science community to rapidly explore effects of input parameters on the analysis results. The system will track sub-results and allow users to browse and query both metadata (e.g., "instrument name") and the complex output data in HDF tables. A new query system that spans modalities (tables, graphs, text, images) will be added. Towards achieving these goals, we will extend the existing BisQue image analysis platform that is widely used for large scale image informatics. The BisQue platform and the associated Materials Research tools will be distributed as open source.