CDS&E: Two- and Three-point Correlations for Large Data Sets with TreeCorr

这位研究人员开发了领先的软件，用于在天文学和其他地方的大型数据集上测量两点相关函数。然而，对于非常大的数据集，当前版本必须跨多个计算机节点拆分，这可能很难正确有效地完成。该项目将增强软件，以使用消息传递接口(MPI)在给定数量的节点上有效地拆分作业。其他软件更改将进一步提高效率，增加三点互相关功能，并包括更准确的剪切估计方法。所有这些修改都是该软件当前用户明确要求的，这将使其对未来数据量更大、精度要求更严格的实验特别有用。这些改进将帮助许多科学家更有效地开展他们的研究。在整个项目中，首席研究员将继续他的活动，以提高其他天文学家的编码技能，帮助他们进行代码设计、算法选择、良好的开发工作流程，以及其他软件开发问题。他的一些课程已经可以通过YouTube免费获得。TreeCorr软件使用了一种非常高效的球树算法来计算相关性，时间为O(N对数N)，这使得它比其他可用的选择要快得多。TreeCorr在单个计算机节点中使用OpenMP，添加MPI将提供最佳的整体效率。其他改进包括更改TreeCorr决定何时停止遍历其树的内部代码，允许任意装箱和任意小的仓位斜率。已经支持O(N Log N)复杂度的三点自相关，但由于复杂的簿记要求，使得能够进行计数-计数-剪切等研究的三点交叉相关已经不可用：这一限制将被取消。新的剪切法，Meta校准，需要额外的计算，但正确地做起来可能又有点复杂。将在TreeCorr中构建正确和高效的元校准代码。该软件已通过PIP下载超过40,000次，并被许多从事当前和即将进行的调查的研究团队使用：他们都将从这项研究的算法进步中受益匪浅。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This researcher has developed the leading software for measuring two-point correlation functions on large data sets in astronomy and elsewhere. However, for very large datasets the current version must be split across multiple computer nodes, which can be complicated to do correctly and efficiently. This project will enhance the software to use the Message Passing Interface (MPI) to split a job efficiently over a given number of nodes. Other software changes will further increase efficiency, add capabilities for three-point cross correlations, and include a more accurate shear estimation method. All of these modifications have been explicitly requested by current users of the software, and will make it especially useful for future experiments with much larger data volumes and tighter accuracy requirements. These improvements will help many scientists to carry out their research more efficiently and effectively. Throughout the project, the principal investigator will continue his campaign to improve the coding skills of other astronomers, helping them with code design, algorithm choices, good development workflow, and other software development issues. Some of his lessons are already freely available through YouTube.The software, TreeCorr, uses a very efficient ball-tree algorithm to compute correlations in O(N log N) time, making it significantly faster than other available options. TreeCorr uses OpenMP within a single computer node, and adding MPI will provide optimum overall efficiency. Other improvements include changing internal code for how TreeCorr decides when to stop traversing its tree, allowing arbitrary binning and arbitrarily small bin slop. Three-point auto-correlations are already supported at O(N log N) complexity, but the three-point cross-correlations that enable studies such as count-count-shear have been unavailable due to the complex bookkeeping required: this limitation will be removed. The new shear method, MetaCalibration, requires additional calculations that again can be somewhat complicated to do correctly. Correct and efficient code for MetaCalibration will be built into TreeCorr. This software has been downloaded via pip over 40,000 times and is used by many research teams working on current and forthcoming surveys: they will all benefit greatly from the algorithmic advances to be made by this study.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.