CSR: Small: A Just-in-Time, Cross-Layer Instrumentation Framework for Diagnosing Performance Problems in Distributed Applications

在数据中心中运行的分布式应用对社会至关重要（例如，购物，银行）。 工程师必须快速诊断和修复数据中心中观察到的问题;然而，这样做极具挑战性。 一个重要的障碍是，工程师必须花费大量的时间和精力来探索什么样的仪器（例如，需要记录关于特定应用程序行为的消息）来提供对新问题的可见性。 为了在这方面提供帮助，该项目将开发一个工具框架，该框架将自动搜索可能的工具选择空间，并启用所需的工具来提供对它的洞察。该项目解决了与创建自动工具框架相关的基本挑战：（a）什么样的算法和算法适合于自动和有效地探索仪器搜索空间？ (b)在框架中需要什么架构支持来实现自动探索？ (c)如何在不显著影响应用程序性能的情况下探索搜索空间？ 该提案将探索基于操作员知识，统计和机器学习的算法的实用性，以探索搜索空间。 它将建立在端到端跟踪的基础上，因为这将使框架能够处理影响不同请求集的问题。该项目将为下一代插装框架的架构提供信息，这些框架需要跟上分布式应用程序日益增长的复杂性。 在评估框架的同时，在流行的开源分布式应用程序中发现的关键问题将提高其鲁棒性。 研究人员将能够利用该项目发布的软件工件来创建新的分布式应用程序管理工具，这些工具利用了框架的独特功能。 他们将能够在研究云中部署框架，以从中获得有价值的工作负载跟踪。 该项目将生成分布式应用程序诊断实践的课程模块。该项目生成的工件，包括框架源代码，工作负载跟踪，仪表应用程序和研究结果，将在https://massopen.cloud和https://www.rajasambasivan.com上免费在线传播。 所有软件工件也将存储在Github中。 所有的文物将可从项目开始至少七年。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Distributed applications running in data centers are critical to society (e.g., for shopping, banking). Engineers must diagnose and fix problems observed in data centers quickly; however, doing so is extremely challenging. A significant hurdle is that engineers must spend significant time and effort exploring what instrumentation (e.g., log messages about specific application behaviors) is needed to provide visibility into a new problem. To assist in this front, this project will develop an instrumentation framework that, in response to a new problem, will automatically search the space of possible instrumentation choices and enable the instrumentation needed to provide insight into it.This project addresses fundamental challenges associated with creating an automatic instrumentation framework: (a) What algorithms and heuristics are suited for automatically and efficiently exploring the instrumentation search space? (b) What architectural support is needed within the framework to enable automatic exploration? (c) How can the search space be explored without significantly impacting application performance? The proposal will explore the utility of algorithms based on operator knowledge, statistics, and machine learning to explore the search space. It will build on end-to-end tracing, as this will enable the framework to work for problems that affect different sets of requests.This project will inform the architecture of next-generation instrumentation frameworks, which are needed to keep pace with the ever-increasing complexity of distributed applications. The critical issues identified in popular open-source distributed applications while evaluating the framework will improve their robustness. Researchers will be able to leverage the software artifacts released by this project to create novel distributed-application-management tools that leverage the framework's unique capabilities. They will be able to deploy the framework in research clouds to obtain valuable workload traces from them. The project will generate course modules on diagnosis practices for distributed applications.The artifacts produced by this project, including framework source code, workload traces, instrumented applications, and research results, will be freely disseminated online at: https://massopen.cloud and https://www.rajasambasivan.com. All software artifacts will be stored in Github as well. All artifacts will be available for a minimum of seven years from the start of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.