Collaborative Research: PPoSS: LARGE: ScaleStuds: Foundations for Correctness Checkability and Performance Predictability of Systems at Scale

鉴于摩尔定律和登纳德缩放的局限性以及不断增长的计算需求，过去十年出现了前所未有的部署规模；众所周知，谷歌运行的集群每个都有数千台机器，苹果总共部署了10万台数据库机器，Netflix运行数十个数据库集群，每个集群有500个节点。这个极端规模分布式系统的时代产生了一类新的故障，“可伸缩性故障”——与规模相关的复杂潜在故障，其症状在大规模部署中显现，但不一定在中小型部署中显现。许多基础研究问题在今天还没有答案。关于正确性：如何通过程序分析发现只有在大规模情况下才会出现的bug ？如何在一台机器上有效地测试和复制不同尺寸的系统规模？如何预防和修复与可伸缩性相关的错误？关于性能：如何推断不同异构设备上的软件性能？如何准确地预测细粒度任务的性能，以减少聚合级别和项目性能对未来架构的不准确性？最后，结合起来：如何为更大的互联生态系统（而不仅仅是单个软件和硬件组件）回答所有这些问题，并最终构建可复制和可验证的下一代系统，这些系统在正确性和性能方面都是大规模的？ScaleStuds项目包括一个由10名研究人员组成的团队，以开发大规模系统的正确性检查性（CC）和性能可预测性（PP）的基础。这个项目的关键原则是“用大来检查大”——用大量的数据、分析、测试、学习、模型和证明来检查大型系统。愿景是建立一个分布式的“CC+ pp认证”软件-软件和硬件交互的生态系统。该项目一次铺设一层“地板”，创造可组合的建筑块（“螺柱”）。该项目首先构建了新的机制，如规模测试平台和统一的软件程序属性数据库，以及公开清晰api的硬件性能配置文件。然后，这些studs可以在软件/硬件堆栈的各个级别上实现多维自动化可伸缩性测试和程序分析以及性能学习和预测。最终，所有这些经验都旨在引导正确和高性能的跨层/服务交互以及未来的设计原则，包括可重复的和按构造验证的开发方法。项目创新包括改进调试、测试、学习和预测方法，以确保经典硬件平台和新兴硬件平台上极端规模系统和应用的正确性、可检查性和性能可预测性；一个统一的软件/硬件属性和配置文件的数据生态系统，通过清晰的api促进自动化分析；多维规模测试框架，支持开发新的大规模单元测试和程序分析；详细的设备分析和观察，以实现大规模的性能学习/预测，并为学习/预测端到端硬件/软件堆栈中的其他设备和层的行为提供经验教训；并最终明确定义当前系统的CC+ pp可认证性和未来可验证/可重复的构建开发方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In light of the limits of Moore's Law and Dennard scaling and the ever increasing computing demand, the last decade has seen unprecedented deployment scales; Google is known to run clusters with thousands of machines each, Apple deploys a total of 100,000 database machines, and Netflix runs tens of database clusters with 500 nodes each. This era of extreme-scale distributed systems has given birth to a new class of faults, "scalability faults" -- complex latent faults that are scale-dependent, whose symptoms surface in large-scale deployments but not necessarily in small/medium-scale deployments. Many fundamental research questions are not answerable today. On correctness: How to detect bugs that only manifest under large scale through program analysis? How to test and reproduce various dimensions of system scales efficiently on one machine? How to prevent and fix scalability-related faults? On performance: How to reason about software performance on various heterogeneous devices? How to accurately predict performance of fine-grained tasks to reduce inaccuracies at the aggregate level and project performance to future architectures? Finally, in combination: How to answer all these questions for the larger connected ecosystem -- not just the individual software and hardware components -- and to eventually build future-generation systems that are reproducible and verifiable by construction with respect to correctness and performance at scale? The ScaleStuds project involves a team of ten researchers to develop the foundations of correctness checkability (CC) and performance predictability (PP) of systems at scale. The key principle of this project is to "check large with large" -- check large-scale systems with a large fleet of data, analysis, tests, learning, models, and proofs. The vision is to build an ecosystem of distributed "CC+PP-certified" software-software and -hardware interactions. The project is paving the vision one "floor" at a time, creating composable building blocks ("the studs"). The project first builds new mechanisms such as a scale-testing platform and a unified database of software program properties and hardware performance profiles exposing clear APIs. These studs then enable multi-dimensional automated scalability tests and program analysis and performance learning and prediction at various levels of the software/hardware stack. Ultimately all of these experiences are intended to lead to correct and performant cross-layer/service interactions and future design principles including reproducible- and verified-by-construction development methods. The project novelties include the advancement of debugging, testing, learning, and prediction methods to ensure correctness checkability and performance predictability of extreme-scale systems and applications both on classical hardware platforms and emerging ones; a unified data ecosystem of software/hardware properties and profiles that facilitates automated analyses via clear APIs; a multi-dimensional scale-testing framework that empowers the development of new large-scale unit-tests and program analysis; detailed device profiling and observation to enable large-scale performance learning/prediction and deliver lessons for learning/predicting the behavior of other devices and layers in an end-to-end hardware/software stack; and ultimately a clear definition of CC+PP-certifiability for today's systems and future verifiable/reproducible-by-construction development methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.