STCI: Middleware for Monitoring and Troubleshooting of Large-Scale Applications on National Cyberinfrastructure

这项提案将使用2009年《美国复苏和再投资法案》(公法111-5)提供的资金授予，并符合2009年3月20日题为《确保负责任地使用复苏法案资金》的白宫备忘录第2节规定的要求。STCI：国家网络基础设施上大规模应用程序监控和故障排除的中间件项目旨在提供健壮且可扩展的工作流监控服务，该服务可用于跟踪基于工作流的应用程序在分布式网络基础设施上执行的进度。还将开发新的异常检测和故障排除服务，以提醒用户应用程序和网络基础设施服务存在的问题，并允许他们快速导航和挖掘应用程序的执行记录。这项工作的基础是为性能信息收集和分发开发一个健壮且可扩展的基础设施。通过这一基础设施流动的信息将存储在高性能档案中，并通过订阅界面分发给感兴趣的实体。将开发三项主要服务：1)在线监控服务，2)基于应用程序和网络基础设施日志的动态挖掘的异常检测服务，以及3)帮助追踪故障来源的故障排除服务。这项工作将使科学家能够快速识别应用程序中的问题，从而减少产生有科学意义的结果所需的时间，从而潜在地提高他们的生产力。这项工作还将使复杂科学工作流程的执行更加透明，从而能够与网络和资源提供商协作，准确估计完成工作所需的总时间，更有效地利用资源，更容易解决端到端的性能问题。更广泛的影响天文学、生物学、地震科学、物理学和其他领域的科学界将立即从拟议的系统中受益。由于该方法依赖于简单、定义良好的日志格式，因此这项工作适用于一系列工作流管理系统以及这些系统的子组件，如作业管理器和数据传输工具。

This proposal will be awarded using funds made available by the American Recovery and Reinvestment Act of 2009 (Public Law 111-5), and meets the requirements established in Section 2 of the White House Memorandum entitled, Ensuring Responsible Spending of Recovery Act Funds, dated March 20, 2009. The STCI: Middleware for Monitoring and Troubleshooting of Large-Scale Applications on National Cyberinfrastructure project aims to provide robust and scalable workflow monitoring services that can be used to track the progress of workflow-based applications as they are executing on the distributed cyberinfrastructure. New anomaly detection and troubleshooting services will also be developed to alert users to problems with the application and cyberinfrastructure services and allow them to quickly navigate and mine the application's execution records. The foundation of this work is the development of a robust and scalable infrastructure for performance information gathering and distribution. Information flowing through this infrastructure will be stored in high-performance archives and distributed to interested entities through subscription interfaces. Three main services will be developed: 1) an online monitoring service, 2) an anomaly detection service based on dynamic mining of application and cyberinfrastructure logs and 3) a troubleshooting service that will help trace the source of a failure.Intellectual Merit This work will potentially increase scientists' productivity by allowing them to quickly identify problems in an application, thus reducing the time it takes to generate scientifically meaningful results. This work will also make the performance of complex scientific workflows more transparent, which will enable the generation of accurate estimates of overall time to completion, more efficient use of resources, and easier resolution of end-to-end performance problems in collaboration with network and resource providers. Broader Impact Scientific communities in astronomy, biology, earthquake science, physics, and others will immediately benefit from the proposed system. Because the approach relies on simple, well-defined logging formats, this work is applicable to a range of workflow management systems as well as sub-components of those systems such as job managers and data transfer tools.