EAGER: Collaborative 3D Materials Science Research in the Cloud

长期以来，先进材料的发现和广泛应用一直受到元素和转化路径极其复杂的组合的挑战，这些组合导致了材料特性的过剩。材料研究项目产生的数据集从几十gb到tb不等，这使得组织、共享和分析变得困难。该项目为材料研究带来了新的大数据管理和分析技术，使科学家能够使用新工具访问以前难以处理的数据集，通过允许共享复杂数据和分析工具来加快创新步伐。该软件将部署在网络上，使世界各地的研究人员可以访问它，共同探索材料的结构。这些工具和方法允许将3维（3- d）和4维（4- d）的实验测量与极端环境中材料的计算建模相结合，例如航空航天发动机部件和高效热电材料的开发。该项目将向多个方向推动基于web的大规模图像和数据分析的边界。首先，在异构计算集群上执行复杂的科学工作流将通过利用虚拟化技术和现代集群计算框架（如Apache Spark）得到简化。我们将比较在实际工作流的不同框架上并行执行策略的开销。其次，我们将为科学工作流程添加来源跟踪和版本控制，包括一个基于web的浏览器，它有助于理解过去的分析运行并提高实验的可重复性。我们将为此目的扩展图形查询系统和图形可视化框架。第三，我们将集成跨参数范围并行运行Dream.3D管道的能力。这将使材料科学界能够快速探索输入参数对分析结果的影响。该系统将跟踪子结果，并允许用户浏览和查询元数据（例如，“仪器名称”）和HDF表中的复杂输出数据。一个新的查询系统，跨越模式（表，图，文本，图像）将被添加。为了实现这些目标，我们将扩展现有的广泛用于大规模图像信息学的BisQue图像分析平台。BisQue平台和相关的材料研究工具将作为开源分发。

The discovery and widespread implementation of advanced materials have long been challenged by the overwhelmingly complex combinations of elements and transformation paths that result in a plethora of material properties. Materials research projects generate datasets that range from a few tens of Gigabytes to terabytes, making it difficult to organize, share and analyze. This project brings new big data management and analysis techniques to materials research, giving scientists access to previously unwieldy datasets with new tools that accelerate the pace of innovation by allowing sharing of both complex data and analysis tools. The software will be deployed on the web, making it accessible to researchers worldwide to collaboratively explore the structure of materials. These tools and methods allow for the integration of experimental measurements in 3 dimensions (3-D) and 4 dimensions (4-D) with computational modeling of materials in extreme environments, such as aerospace engine components and the development of high efficiency thermoelectric materials.The project will push the boundary of large-scale web-based image and data analysis in multiple directions. First, the execution of complex scientific workflows on heterogeneous compute clusters will be simplified by exploiting virtualization techniques and modern cluster computing frameworks such as Apache Spark. We will compare overheads for parallel execution strategies on different frameworks for realistic workflows. Second, we will add provenance tracking and versioning for scientific workflows including a web-based browser that aids in making sense of past analysis runs and improves repeatability of experiments. We will extend graph query systems and graph visualization frameworks for this purpose. Third, we will integrate the capability to run Dream.3D pipelines in parallel across parameter ranges. This will enable the Materials Science community to rapidly explore effects of input parameters on the analysis results. The system will track sub-results and allow users to browse and query both metadata (e.g., "instrument name") and the complex output data in HDF tables. A new query system that spans modalities (tables, graphs, text, images) will be added. Towards achieving these goals, we will extend the existing BisQue image analysis platform that is widely used for large scale image informatics. The BisQue platform and the associated Materials Research tools will be distributed as open source.