SI2-SSI: Integrating the NIMBLE Statistical Algorithm Platform with Advanced Computational Tools and Analysis Workflows

该项目开发的软件将使科学家能够从复杂的数据中学到更多东西，并更容易地分享新的分析方法。越来越多领域的科学家致力于从庞大而复杂的数据集中得出可靠的结论。这些领域包括环境生物学、政治学、教育研究、大气和海洋科学、气候科学等许多领域。数据可能是复杂的，因为许多相关变量被测量和/或因为一些测量彼此不独立。当一些变量在一段时间内被重复测量时，或当在附近地点进行测量时，或当对一组相关的个人进行测量时，或由于这些原因和其他类似原因的组合，就可能出现非独立性。对于这种情况，已经开发了通用的统计方法，使研究人员能够针对每个数据集进行分析，以便说明数据之间的关系。鉴于有限数据中固有的不确定性，这些方法依赖于计算机算法来探索可能的结论范围。在这些一般方法中，有许多已经并将继续发展的具体方法。因此，出现了一个重大的软件缺口：许多新的和不断发展的方法不容易被广泛的科学家应用，因为还没有一个软件框架使它们易于编程和传播。该项目将支持继续开发Nimble软件，以帮助填补这一空白。因此，科学家将能够更灵活地使用计算分析方法，更容易地组合和比较不同的算法，将这些算法集成到其他软件工作流中，并获得更好的计算性能。现有的用于分层统计模型和算法的灵活框架包括模型规范语言、用于在R统计环境中编程模型通用算法的语言、以及生成、编译和接口到特定于模型和算法的C++以高效执行的编译器。这些使得诸如马尔可夫链蒙特卡罗、顺序蒙特卡罗和许多相关方法的一般实现和传播成为可能。在这个项目中，将对Nimble进行扩展和推广，使其更加强大和灵活，从而能够在各种软件工作流中使用。对Nimble核心能力的扩展将包括利用生成的C++中的自动区分和并行化，增强其现有的线性代数能力，更有效地实施大型统计模型，包括那些具有结构不确定性的模型，如潜在的组成员资格，以及对统计建模语言的扩展。促进灵活生成的模型和算法与其他软件集成的增强功能将包括生成独立的可执行文件、生成明确定义的应用程序编程人员接口(如供Python使用)、从算法代码调用用户提供的库的功能、通过JSON和NetCDF等标准格式加载和保存数据的功能，以及将灵活组件分离到不同的包中。该项目将包括大量的外展、培训和用户社区发展。这些活动将包括开发人口和生态系统生态学、海洋学、气候科学、政治学和教育等领域的用例。它们还将包括讲习班、用户会议、主要用户访问和培训材料。这一奖项由高级数字基础设施部颁发，由美国国家科学基金会数学和物理科学局(数学科学部)联合支持。

The software developed in this project will enable scientists to learn more from complex data and to share new analysis methods more easily. Increasingly, scientists in many fields aim to draw sound conclusions from large and complex data sets. Such fields include environmental biology, political science, education research, atmospheric and oceanic science, climate science, and many others. Data may be complex because many related variables are measured and/or because some measurements are not independent from each other. Non-independence can arise when some variables are measured repeatedly through time; or when measurements are made at nearby locations; or when measurements are made on groups of related individuals; or for a combination of those and other similar reasons. For such cases, general statistical methods have been developed to allow researchers to tailor their analysis to each data set in order to account for the relationships among the data. Such methods rely on computer algorithms to explore the range of possible conclusions given the uncertainties inherent in limited data. Within those general methods there are many varieties of specific approaches that have been and continue to be developed. Thus, a major software gap has emerged: Many new and evolving methods are not easily available for application by a wide range of scientists because there has not been a software framework that makes them easy to program and disseminate. This project will support continued development of the NIMBLE software to help fill that gap. As a result, scientists will be able to use computational analysis methods more flexibly, to combine and compare different algorithms more easily, to integrate such algorithms into other software workflows, and to gain better computational performance. This will enable more advanced and more routine use of some modern computational methods for analyzing complex data.The existing NIMBLE framework for hierarchical statistical models and algorithms comprises a model specification language, a language for programming model-generic algorithms within the R statistical environment, and a compiler that generates, compiles and interfaces to model- and algorithm-specific C++ for efficient execution. These enable general implementation and dissemination of methods such as Markov chain Monte Carlo, sequential Monte Carlo, and many related methods. In this project NIMBLE will be extended and generalized to be more powerful and flexible, enabling use in a variety of software workflows. Extensions to NIMBLE's core capabilities will include harnessing automatic differentiation and parallelization in generated C++, enhancements to its existing linear algebra capabilities, more efficient implementation of large statistical models including those with structural uncertainty such as latent group membership, and extensions to the statistical modeling language. Enhancements to facilitate integration of NIMBLE-generated models and algorithms with other software will include generation of stand-alone executables, generation of clearly defined application-programmer interfaces such as for use by Python, features to call user-provided libraries from algorithm code, features to load and save data via standard formats such as JSON and NetCDF, and separation of NIMBLE components into distinct packages. The project will include substantial outreach, training, and user community development. These activities will include development of uses cases in fields such as population and ecosystem ecology, oceanography, climate science, political science, and education. They will also include workshops, user meetings, key-user visits, and training material. This award by the Advanced Cyberinfrastructure Division is jointly supported by the NSF Directorate for Mathematical and Physical Sciences (Division of Mathematical Sciences).