CSR-AES: Troubleshooting Large Scale Computing Grids with Machine Learning Techniques

CSR-AES：利用机器学习技术对大规模计算网格进行故障排除

• 批准号: 0720813
• 负责人: Nitesh Chawla
• 金额: $ 3万
• 依托单位: University of Notre Dame
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2007
• 资助国家: 美国
• 起止时间: 2007-08-01 至 2008-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0720813&HistoricalAwards=false
• 关键词: CSR; AES; Troubleshooting; Large; Scale

Both users and administrators of computing grids are presented with enormous challenges in debugging and troubleshooting. Diagnosing a problem with one application on one machine is hard enough, but diagnosing problems in workloads of millions of jobs running on thousands of machines is a problem of a new order of magnitude. Suppose that a user submits one million jobs to a grid, only to discover some time later that half of them have failed. Each individual failure could be the manifestation of one of many kinds of error: a job specification error, a machine configuration error, a transient system state, and many others. It does little use to investigate any one instance of failure. Rather, users of large scale systems need tools that describe the overall situation, indicating what problems are commonplace versus occasional, and which are deterministic versus random. Machine learning techniques can be used to debug these kinds of problems in large scale systems. The proposal poses the following research questions: What sort of failures is most common in grids? What data must be collected to identify these failures? What innovations in machine learning algorithms are required to be successful in this domain? This project will generate new understanding by collecting large amounts of production data from TeraGrid and OSG, developing novel analysis techniques, and working closely with end users to produce useful diagnoses. The results will provide both new understanding of complex large scale computer systems, as well as innovations in machine learning algorithms to tackle such scenarios.

计算网格的用户和管理员在调试和故障排除方面都面临着巨大的挑战。诊断一台机器上的一个应用程序的问题已经很困难了，但是诊断在数千台机器上运行的数百万个作业的工作负载中的问题是一个新的数量级的问题。假设用户向网格提交了一百万个作业，一段时间后才发现其中一半失败了。每个单独的故障都可能是多种错误之一的表现：作业规范错误、机器配置错误、瞬态系统状态以及许多其他错误。调查任何一个失败的例子都没有什么用处。相反，大规模系统的用户需要能够描述整体情况的工具，指出哪些问题是常见的，哪些是偶然的，哪些是确定性的，哪些是随机的。机器学习技术可用于在大型系统中调试这类问题。该提案提出了以下研究问题：电网中最常见的故障类型是什么？必须收集哪些数据来识别这些故障？要在这个领域取得成功，机器学习算法需要哪些创新？该项目将通过从TeraGrid和OSG收集大量生产数据，开发新的分析技术，并与最终用户密切合作，产生有用的诊断，从而产生新的认识。研究结果将提供对复杂的大规模计算机系统的新理解，以及解决此类场景的机器学习算法的创新。