CC* Integration: Service Analysis and Network Diagnosis (SAND)

科学越来越多地成为一项“团队运动”，项目不太可能由单一的研究人员或大学团体进行，而是由多个机构合作，跨越许多大学和国家实验室。随着这些协作变得数据密集型，连接其分布式科学平台的高性能网络链接变得越来越重要。虽然在所有科学中都是如此，但一些合作，如欧洲核子研究中心的大型强子对撞机(LHC)的合作，今天在全球网络中传输了数百万GB的数据，并预计在未来十年内将增长数量级。如果没有高性能的网络和高效的数据传输和访问服务，科学研究的时间将大大减少。在这种情况下，高速、长途网络在面对适度的错误率时，其性能会以惊人的速度下降。因此，网络工程师和研究人员使用复杂的工具来监控网络和传输服务。如果没有聚合和关联网络测试、性能测量和应用程序响应的方法，它们充其量只能揭示整个问题的一小部分。该项目侧重于更好地组合、可视化和分析不同的网络监控和服务日志记录数据的技术，为依赖网络的工程师和科学家提供了一幅至关重要的全面图景。这将允许更快地定位和解决问题，减少进行科学研究的时间。“CC*集成：服务分析和网络诊断(SAND)”项目汇集了一支经验丰富的团队，他们在大规模科学的广域数据传输方面已经工作了十多年。该项目开发了一个网络监控档案和分析平台SAND-NMA，它将广泛使用的数据分析工具(如ElasticSearch、Kibana和JupyterLab)与基础设施组件(PerformSONAR、HTCondor)和应用传感器集成在一起。来自不同来源的数据被发布到消息传递总线，提供描述全球范围内研究平台的性能的低延迟指标。探索性工作旨在为工程师提供实用工具来识别和定位问题，并执行分析以了解网络的长期演变和更高级别的服务性能。编程接口允许外部网络基础设施(如工作流管理系统)将网络监控馈送纳入其决策引擎。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Increasingly, science has become a "team sport" with projects less likely to be conducted by a single investigator or university group but with multi-institution collaborations spanning many universities and national laboratories. As these collaborations have become data-intensive, highly-performant network links connecting their distributed science platforms have become increasingly important. While true across all science, some collaborations, like those at the Large Hadron Collider (LHC) at CERN, transfer millions of gigabytes across global networks today and anticipate orders of magnitude increases within the next decade. Without performant networks and efficient data transfer and access services, the time to science will be greatly compromised. In this context, high-speed, long-distance networks see their performance degrade surprisingly quickly in the face of modest error rates. Accordingly, network engineers and researchers use sophisticated tools to monitor the network and transfer services. Without a means to aggregate and correlate network tests, performance measurements, and application response, they at best can only reveal a small piece of the overall problem. This project focuses on techniques that better combine, visualize, and analyze disparate network monitoring and service logging data, providing a comprehensive picture critical to the engineers and scientists relying on the network. This will allow problems to be located and fixed more quickly, reducing the time to science. The "CC* Integration: Service Analysis and Network Diagnosis (SAND)" project brings together an experienced team that has been working for more than a decade on wide area data transfers for large scale science. The project develops a network monitoring archive and analytics platform, SAND-NMA, which integrates widely used data analytics tools (such as ElasticSearch, Kibana and JupyterLab) with infrastructure components (perfSONAR, HTCondor) and application sensors. Data from disparate sources are published to a messaging bus providing low-latency metrics describing the performance of research platforms on a global scale. Exploratory work is performed to provide engineers with pragmatic tools to identify and locate problems and perform analytics to understand the long-term evolution of network and higher level service performance. Programming interfaces allow external cyberinfrastructure (such as workflow management systems) to incorporate the network monitoring feeds into their decision making engines.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.