Collaborative Research: PPoSS: LARGE: ScaleStuds: Foundations for Correctness Checkability and Performance Predictability of Systems at Scale

鉴于摩尔定律和Dennard扩展的局限性以及不断增长的计算需求，过去十年出现了前所未有的部署规模；众所周知，谷歌运行的集群每个有数千台机器，苹果总共部署了10万台数据库机器，Netflix运行着数十个数据库集群，每个集群有500个节点。这个极端规模分布式系统的时代催生了一类新的故障--“可伸缩性故障”--依赖于规模的复杂的潜在故障，其症状在大规模部署中显露出来，但不一定在小型/中型部署中。许多基础研究问题在今天是无法回答的。关于正确性：如何通过程序分析来检测只有在大规模情况下才会出现的错误？如何在一台机器上高效地测试和再现各种规模的系统？如何预防和修复与可伸缩性相关的故障？关于性能：如何推断各种不同类型设备上的软件性能？如何准确预测细粒度任务的性能，以减少聚合级别的不准确性和未来架构的项目性能？最后，结合：如何为更大的互联生态系统回答所有这些问题--而不仅仅是单个软件和硬件组件--并最终构建可复制和可通过构建在规模上的正确性和性能方面进行验证的未来一代系统？ScaleStuds项目涉及一个由10名研究人员组成的团队，以开发规模化系统的正确性可检查性(CC)和性能可预测性(PP)的基础。这个项目的关键原则是“以大查大”--用大量的数据、分析、测试、学习、模型和证明来检查大型系统。我们的愿景是建立一个分布式的“CC PP认证”软件-软件-硬件交互的生态系统。该项目正在一次铺设一层楼的愿景，创造出可组合的积木(“螺柱”)。该项目首先建立了新的机制，如规模测试平台和一个统一的软件程序属性和硬件性能配置文件数据库，以公开明确的API。然后，这些STUD能够在软件/硬件堆栈的不同级别上实现多维自动可伸缩性测试和程序分析以及性能学习和预测。归根结底，所有这些经验都旨在产生正确和有效的跨层/服务交互以及未来的设计原则，包括可重复使用和通过构建验证的开发方法。该项目的创新之处包括：调试、测试、学习和预测方法的进步，以确保在传统硬件平台和新兴硬件平台上极端规模的系统和应用程序的正确性、可检查性和性能可预测性；统一的软件/硬件属性和配置文件数据生态系统，通过清晰的API促进自动化分析；多维规模测试框架，支持新的大规模单元测试和程序分析的开发；详细的设备配置和观察，以实现大规模性能学习/预测，并提供学习/预测端到端硬件/软件堆栈中其他设备和层的行为的经验教训；并最终明确定义CC PP--今天的系统和未来可验证/可重复建设的开发方法的可认证性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，认为值得支持。

In light of the limits of Moore's Law and Dennard scaling and the ever increasing computing demand, the last decade has seen unprecedented deployment scales; Google is known to run clusters with thousands of machines each, Apple deploys a total of 100,000 database machines, and Netflix runs tens of database clusters with 500 nodes each. This era of extreme-scale distributed systems has given birth to a new class of faults, "scalability faults" -- complex latent faults that are scale-dependent, whose symptoms surface in large-scale deployments but not necessarily in small/medium-scale deployments. Many fundamental research questions are not answerable today. On correctness: How to detect bugs that only manifest under large scale through program analysis? How to test and reproduce various dimensions of system scales efficiently on one machine? How to prevent and fix scalability-related faults? On performance: How to reason about software performance on various heterogeneous devices? How to accurately predict performance of fine-grained tasks to reduce inaccuracies at the aggregate level and project performance to future architectures? Finally, in combination: How to answer all these questions for the larger connected ecosystem -- not just the individual software and hardware components -- and to eventually build future-generation systems that are reproducible and verifiable by construction with respect to correctness and performance at scale? The ScaleStuds project involves a team of ten researchers to develop the foundations of correctness checkability (CC) and performance predictability (PP) of systems at scale. The key principle of this project is to "check large with large" -- check large-scale systems with a large fleet of data, analysis, tests, learning, models, and proofs. The vision is to build an ecosystem of distributed "CC+PP-certified" software-software and -hardware interactions. The project is paving the vision one "floor" at a time, creating composable building blocks ("the studs"). The project first builds new mechanisms such as a scale-testing platform and a unified database of software program properties and hardware performance profiles exposing clear APIs. These studs then enable multi-dimensional automated scalability tests and program analysis and performance learning and prediction at various levels of the software/hardware stack. Ultimately all of these experiences are intended to lead to correct and performant cross-layer/service interactions and future design principles including reproducible- and verified-by-construction development methods. The project novelties include the advancement of debugging, testing, learning, and prediction methods to ensure correctness checkability and performance predictability of extreme-scale systems and applications both on classical hardware platforms and emerging ones; a unified data ecosystem of software/hardware properties and profiles that facilitates automated analyses via clear APIs; a multi-dimensional scale-testing framework that empowers the development of new large-scale unit-tests and program analysis; detailed device profiling and observation to enable large-scale performance learning/prediction and deliver lessons for learning/predicting the behavior of other devices and layers in an end-to-end hardware/software stack; and ultimately a clear definition of CC+PP-certifiability for today's systems and future verifiable/reproducible-by-construction development methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.