STCI: Middleware for Monitoring and Troubleshooting of Large-Scale Applications on National Cyberinfrastructure

该提案将使用2009年美国复苏和再投资法案（公法111-5）提供的资金，并符合2009年3月20日白宫备忘录第2节的要求，题为“确保负责任的复苏法案资金支出”。STCI：国家网络基础设施大规模应用监控和故障排除中间件项目旨在提供健壮且可扩展的工作流监控服务，可用于跟踪基于工作流的应用程序在分布式网络基础设施上执行的进度。还将开发新的异常检测和故障排除服务，以提醒用户注意应用程序和网络基础设施服务的问题，并允许他们快速导航和挖掘应用程序的执行记录。这项工作的基础是开发用于性能信息收集和分发的健壮且可扩展的基础设施。流经此基础设施的信息将存储在高性能存档中，并通过订阅接口分发给感兴趣的实体。将开发三种主要服务：1)在线监测服务；2)基于应用程序和网络基础设施日志动态挖掘的异常检测服务；3)有助于追踪故障来源的故障排除服务。这项工作将允许科学家快速识别应用程序中的问题，从而减少产生有科学意义的结果所需的时间，从而潜在地提高科学家的生产力。这项工作还将使复杂的科学工作流程的性能更加透明，从而能够准确估计完成的总时间，更有效地利用资源，并更容易地解决与网络和资源提供商合作的端到端性能问题。更广泛的影响天文学、生物学、地震科学、物理学和其他领域的科学界将立即受益于拟议的系统。由于该方法依赖于简单的、定义良好的日志格式，因此这项工作适用于一系列工作流管理系统以及这些系统的子组件，例如作业管理器和数据传输工具。

This proposal will be awarded using funds made available by the American Recovery and Reinvestment Act of 2009 (Public Law 111-5), and meets the requirements established in Section 2 of the White House Memorandum entitled, Ensuring Responsible Spending of Recovery Act Funds, dated March 20, 2009. The STCI: Middleware for Monitoring and Troubleshooting of Large-Scale Applications on National Cyberinfrastructure project aims to provide robust and scalable workflow monitoring services that can be used to track the progress of workflow-based applications as they are executing on the distributed cyberinfrastructure. New anomaly detection and troubleshooting services will also be developed to alert users to problems with the application and cyberinfrastructure services and allow them to quickly navigate and mine the application's execution records. The foundation of this work is the development of a robust and scalable infrastructure for performance information gathering and distribution. Information flowing through this infrastructure will be stored in high-performance archives and distributed to interested entities through subscription interfaces. Three main services will be developed: 1) an online monitoring service, 2) an anomaly detection service based on dynamic mining of application and cyberinfrastructure logs and 3) a troubleshooting service that will help trace the source of a failure.Intellectual Merit This work will potentially increase scientists' productivity by allowing them to quickly identify problems in an application, thus reducing the time it takes to generate scientifically meaningful results. This work will also make the performance of complex scientific workflows more transparent, which will enable the generation of accurate estimates of overall time to completion, more efficient use of resources, and easier resolution of end-to-end performance problems in collaboration with network and resource providers. Broader Impact Scientific communities in astronomy, biology, earthquake science, physics, and others will immediately benefit from the proposed system. Because the approach relies on simple, well-defined logging formats, this work is applicable to a range of workflow management systems as well as sub-components of those systems such as job managers and data transfer tools.