CC* Integration: Service Analysis and Network Diagnosis (SAND)

CC* 集成：服务分析和网络诊断 (SAND)

• 批准号: 1827116
• 负责人: Brian Bockelman
• 金额: $ 49.97万
• 依托单位: University of Nebraska-Lincoln
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1827116&HistoricalAwards=false
• 关键词: CC; Integration; Service; Analysis; Network

Increasingly, science has become a "team sport" with projects less likely to be conducted by a single investigator or university group but with multi-institution collaborations spanning many universities and national laboratories. As these collaborations have become data-intensive, highly-performant network links connecting their distributed science platforms have become increasingly important. While true across all science, some collaborations, like those at the Large Hadron Collider (LHC) at CERN, transfer millions of gigabytes across global networks today and anticipate orders of magnitude increases within the next decade. Without performant networks and efficient data transfer and access services, the time to science will be greatly compromised. In this context, high-speed, long-distance networks see their performance degrade surprisingly quickly in the face of modest error rates. Accordingly, network engineers and researchers use sophisticated tools to monitor the network and transfer services. Without a means to aggregate and correlate network tests, performance measurements, and application response, they at best can only reveal a small piece of the overall problem. This project focuses on techniques that better combine, visualize, and analyze disparate network monitoring and service logging data, providing a comprehensive picture critical to the engineers and scientists relying on the network. This will allow problems to be located and fixed more quickly, reducing the time to science. The "CC* Integration: Service Analysis and Network Diagnosis (SAND)" project brings together an experienced team that has been working for more than a decade on wide area data transfers for large scale science. The project develops a network monitoring archive and analytics platform, SAND-NMA, which integrates widely used data analytics tools (such as ElasticSearch, Kibana and JupyterLab) with infrastructure components (perfSONAR, HTCondor) and application sensors. Data from disparate sources are published to a messaging bus providing low-latency metrics describing the performance of research platforms on a global scale. Exploratory work is performed to provide engineers with pragmatic tools to identify and locate problems and perform analytics to understand the long-term evolution of network and higher level service performance. Programming interfaces allow external cyberinfrastructure (such as workflow management systems) to incorporate the network monitoring feeds into their decision making engines.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学越来越成为一项“团队运动”，项目不太可能由单一的研究人员或大学团体进行，而是由跨越许多大学和国家实验室的多机构合作进行。 随着这些合作变得数据密集型，连接其分布式科学平台的高性能网络链接变得越来越重要。 虽然在所有科学领域都是如此，但一些合作，如欧洲核子研究中心的大型强子对撞机（LHC），今天在全球网络中传输了数百万千兆字节，并预计在未来十年内会增加几个数量级。如果没有高性能的网络和高效的数据传输和访问服务，科学研究的时间将大大缩短。 在这种情况下，高速、长距离网络在面对适度的错误率时，其性能会以惊人的速度下降。 因此，网络工程师和研究人员使用复杂的工具来监控网络和传输服务。 如果没有一种方法来聚合和关联网络测试、性能测量和应用程序响应，它们充其量只能揭示整个问题的一小部分。 该项目的重点是更好地联合收割机，可视化和分析不同的网络监控和服务日志数据的技术，为依赖网络的工程师和科学家提供一个全面的画面。 这将允许更快地定位和修复问题，减少科学研究的时间。“CC* 集成：服务分析和网络诊断（SAND）”项目汇集了一个经验丰富的团队，他们已经在大规模科学的广域数据传输方面工作了十多年。 该项目开发了一个网络监控归档和分析平台SAND-NMA，它将广泛使用的数据分析工具（如ElasticSearch，Kibana和QuanterLab）与基础设施组件（perfSONAR，HTCondor）和应用传感器集成在一起。来自不同来源的数据发布到消息总线，提供描述全球范围内研究平台性能的低延迟指标。进行探索性工作，为工程师提供实用的工具，以识别和定位问题，并进行分析，以了解网络的长期发展和更高级别的服务性能。 编程接口允许外部网络基础设施（如工作流管理系统）将网络监控馈送纳入其决策引擎。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

WLCG Networks: Update on Monitoring and Analytics

WLCG Network：监控和分析更新

• DOI: 10.1051/epjconf/202024507053
• 发表时间: 2020
• 期刊: EPJ Web of Conferences
• 作者: Babik, Marian;McKee, Shawn;Andrade, Pedro;Bockelman, Brian Paul;Gardner, Robert;Fajardo Hernandez, Edgar Mauricio;Martelli, Edoardo;Vukotic, Ilija;Weitzel, Derek;Zvada, Marian
• 通讯作者: Zvada, Marian