Collaborative Research: PPoSS: LARGE: ScaleStuds: Foundations for Correctness Checkability and Performance Predictability of Systems at Scale

鉴于摩尔定律和登纳德缩放定律的限制以及不断增长的计算需求，过去十年出现了前所未有的部署规模；众所周知，Google 运行着每个包含数千台机器的集群，Apple 总共部署了 100,000 台数据库机器，Netflix 运行着数十个数据库集群，每个集群有 500 个节点。 这个超大规模分布式系统时代催生了一类新的故障，即“可扩展性故障”——与规模相关的复杂潜在故障，其症状在大规模部署中显现，但不一定在中小型部署中显现。 今天，许多基础研究问题都无法回答。 关于正确性：如何通过程序分析来检测大规模下才会出现的错误？如何在一台机器上高效地测试和重现系统规模的各个维度？如何预防和修复与可扩展性相关的故障？ 关于性能：如何推理各种异构设备上的软件性能？如何准确预测细粒度任务的性能，以减少总体水平的不准确性以及未来架构的项目性能？ 最后，综合起来：如何为更大的互联生态系统（而不仅仅是单个软件和硬件组件）回答所有这些问题，并最终构建可通过构建在规模正确性和性能方面可复制和可验证的下一代系统？ ScaleStuds 项目由十名研究人员组成的团队负责开发大规模系统的正确性可检查性 (CC) 和性能可预测性 (PP) 的基础。该项目的关键原则是“用大来检查”——用大量数据、分析、测试、学习、模型和证明来检查大型系统。 愿景是建立一个分布式“CC PP 认证”软硬件交互的生态系统。 该项目一次将愿景铺成一层“地板”，创建可组合的构建块（“螺柱”）。 该项目首先构建了新的机制，例如规模测试平台以及公开清晰 API 的软件程序属性和硬件性能配置文件的统一数据库。 然后，这些螺柱可以在软件/硬件堆栈的各个级别上实现多维自动化可扩展性测试和程序分析以及性能学习和预测。 最终，所有这些经验都旨在带来正确且高性能的跨层/服务交互和未来的设计原则，包括可重复和可验证的构建开发方法。 该项目的新颖之处包括调试、测试、学习和预测方法的进步，以确保经典硬件平台和新兴硬件平台上极端规模系统和应用程序的正确性可检查性和性能可预测性；一个由软件/硬件属性和配置文件组成的统一数据生态系统，可通过清晰的 API 促进自动分析；多维规模测试框架，支持开发新的大规模单元测试和程序分析；详细的设备分析和观察，以实现大规模性能学习/预测，并为学习/预测端到端硬件/软件堆栈中其他设备和层的行为提供课程；并最终对当今系统的 CC PP 可验证性和未来可验证/可重现的构建开发方法进行明确定义。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

In light of the limits of Moore's Law and Dennard scaling and the ever increasing computing demand, the last decade has seen unprecedented deployment scales; Google is known to run clusters with thousands of machines each, Apple deploys a total of 100,000 database machines, and Netflix runs tens of database clusters with 500 nodes each. This era of extreme-scale distributed systems has given birth to a new class of faults, "scalability faults" -- complex latent faults that are scale-dependent, whose symptoms surface in large-scale deployments but not necessarily in small/medium-scale deployments. Many fundamental research questions are not answerable today. On correctness: How to detect bugs that only manifest under large scale through program analysis? How to test and reproduce various dimensions of system scales efficiently on one machine? How to prevent and fix scalability-related faults? On performance: How to reason about software performance on various heterogeneous devices? How to accurately predict performance of fine-grained tasks to reduce inaccuracies at the aggregate level and project performance to future architectures? Finally, in combination: How to answer all these questions for the larger connected ecosystem -- not just the individual software and hardware components -- and to eventually build future-generation systems that are reproducible and verifiable by construction with respect to correctness and performance at scale? The ScaleStuds project involves a team of ten researchers to develop the foundations of correctness checkability (CC) and performance predictability (PP) of systems at scale. The key principle of this project is to "check large with large" -- check large-scale systems with a large fleet of data, analysis, tests, learning, models, and proofs. The vision is to build an ecosystem of distributed "CC+PP-certified" software-software and -hardware interactions. The project is paving the vision one "floor" at a time, creating composable building blocks ("the studs"). The project first builds new mechanisms such as a scale-testing platform and a unified database of software program properties and hardware performance profiles exposing clear APIs. These studs then enable multi-dimensional automated scalability tests and program analysis and performance learning and prediction at various levels of the software/hardware stack. Ultimately all of these experiences are intended to lead to correct and performant cross-layer/service interactions and future design principles including reproducible- and verified-by-construction development methods. The project novelties include the advancement of debugging, testing, learning, and prediction methods to ensure correctness checkability and performance predictability of extreme-scale systems and applications both on classical hardware platforms and emerging ones; a unified data ecosystem of software/hardware properties and profiles that facilitates automated analyses via clear APIs; a multi-dimensional scale-testing framework that empowers the development of new large-scale unit-tests and program analysis; detailed device profiling and observation to enable large-scale performance learning/prediction and deliver lessons for learning/predicting the behavior of other devices and layers in an end-to-end hardware/software stack; and ultimately a clear definition of CC+PP-certifiability for today's systems and future verifiable/reproducible-by-construction development methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.