CSR-AES: Troubleshooting Large Scale Computing Grids with Machine Learning Techniques

计算网格的用户和管理员在调试和故障排除方面都面临着巨大的挑战。在一台机器上诊断一个应用程序的问题已经够难的了，但在数千台机器上运行的数百万个作业的工作负载中诊断问题是一个新的数量级的问题。假设用户向网格提交了一百万个作业，但过了一段时间才发现其中一半作业失败了。每个单独的故障可能是多种错误中的一种的表现形式：作业规范错误、机器配置错误、瞬时系统状态以及许多其他错误。它对调查任何一个失败的例子都没有什么用处。相反，大型系统的用户需要工具来描述总体情况，指出哪些问题是常见的还是偶然的，哪些是确定性的而不是随机的。机器学习技术可以用来调试大规模系统中的这类问题。该提案提出了以下研究问题：电网中最常见的故障类型是什么？必须收集哪些数据才能识别这些故障？机器学习算法需要哪些创新才能在这一领域取得成功？该项目将通过从TeraGrid和OSG收集大量生产数据，开发新的分析技术，并与最终用户密切合作，产生有用的诊断，从而产生新的理解。这些结果将提供对复杂大规模计算机系统的新理解，以及机器学习算法的创新，以应对此类场景。

Both users and administrators of computing grids are presented with enormous challenges in debugging and troubleshooting. Diagnosing a problem with one application on one machine is hard enough, but diagnosing problems in workloads of millions of jobs running on thousands of machines is a problem of a new order of magnitude. Suppose that a user submits one million jobs to a grid, only to discover some time later that half of them have failed. Each individual failure could be the manifestation of one of many kinds of error: a job specification error, a machine configuration error, a transient system state, and many others. It does little use to investigate any one instance of failure. Rather, users of large scale systems need tools that describe the overall situation, indicating what problems are commonplace versus occasional, and which are deterministic versus random. Machine learning techniques can be used to debug these kinds of problems in large scale systems. The proposal poses the following research questions: What sort of failures is most common in grids? What data must be collected to identify these failures? What innovations in machine learning algorithms are required to be successful in this domain? This project will generate new understanding by collecting large amounts of production data from TeraGrid and OSG, developing novel analysis techniques, and working closely with end users to produce useful diagnoses. The results will provide both new understanding of complex large scale computer systems, as well as innovations in machine learning algorithms to tackle such scenarios.