CSR: Small: BaaS: Benchmarking-As-A-Service for Improved Visibility, Diagnosis, and Control of Modern Data Analytics Platforms

系统基准是应用程序开发人员和系统管理员用来帮助他们决定如何最好地执行广泛的系统管理任务的关键技术。这些任务的范围从设计时方面(如挑选应用程序使用哪个系统)，到测试和调优应用程序和系统的所有部署前方面，以及生产方面(如可靠性保证和问题诊断)。然而，今天的现实是，应用程序开发人员和系统管理员很少满意地利用基准测试的巨大潜力。基准测试非常耗费人力物力。用户可能不知道或不遵循正确的基准测试方法。通常，在没有充分意识到这样做的影响的情况下，偷工减料是为了减少时间或成本。基准测试结果的可重复性和重用性通常是无法实现的。基准工作的结果可能会被误解，导致不正确或次优的决策。该项目将新的研究与最新的技术趋势相结合，以使不同的用户社区能够获得基准的力量。该项目正在构建一个基准即服务(Baas)平台，该平台在基础设施即服务或平台即服务的云平台上运行。BAAS提供了四种集成服务来简化整个基准测试过程，同时解决了一些新的技术挑战：1.设计：帮助用户创建最符合其特定基准测试需求的工作负载和数据，将基准测试设计的概念用作搜索问题；2.执行：帮助用户以及时且经济高效的方式在适当的数据、系统配置和硬件和软件资源上运行选定的工作负载；3.解释：帮助用户正确解释基准测试的结果，因为许多在系统上运行基准测试的用户远不是这些系统的专家(由于系统监控数据的大量、多样和速度，在这里会产生许多挑战)；4、构建：帮助用户在BaaS之上构建新的系统管理服务。例如，Baas支持自动调优在Amazon Web Services和SQL Azure等云平台上运行的生产数据库实例。Baas正在制作原型，并将寻求在现实世界环境中部署它的机会。该原型将用于评估BAAS在提高人类和系统生产率以及避免涉及关键任务系统的错误决策方面的变革能力。

System benchmarking is a critical technique used by application developers and system administrators to help them decide how best to perform a wide spectrum of system management tasks. These tasks range from design-time aspects such as picking which system to use for an application, to all pre-deployment aspects of testing and tuning the applications and system, and to production aspects such as reliability assurance and problem diagnosis. However, the reality today is that the massive potential of benchmarking is seldom harnessed satisfactorily by application developers and system administrators. Benchmarking is very human-effort-intensive. Users may not be aware of or follow correct benchmarking methodology. Often corners are cut to reduce either time or cost without full awareness of the implications of doing so. Repeatability and reuse of benchmarking results are often unattainable. The results of a benchmarking effort may be misinterpreted, leading to incorrect or suboptimal decisions.This project combines new research with recent technology trends in order to make the power of benchmarking accessible to a diverse community of users. The project is building a Benchmarking-as-a-Service (BaaS) platform that runs on top of an Infrastructure-as-a-Service or Platform-as-a-Service cloud platform. A BaaS provides four integrated services to simplify the overall benchmarking process, while addressing a number of novel technical challenges: 1. Design: Helps the user create the workloads and data that best meet her specific benchmarking needs, using the concept of benchmark design as a search problem; 2. Execute: Helps the user run the selected workloads on appropriate data, system configurations, and hardware and software resources in a timely and cost-efficient manner; 3. Interpret: Helps the user interpret the results of the benchmark correctly since many users who run benchmarks on systems are far from being experts on these systems (many challenges arise here due to the large volume, variety, and velocity of system monitoring data); and 4. Build: Helps the user build new system management services on top of the BaaS. For example, BaaS enables the automatic tuning of production database instances running on cloud platforms like Amazon Web Services and SQL Azure. BaaS is being prototyped and opportunities for deploying it in real-world settings will be pursued. The prototype will be used to evaluate the transformative capabilities of BaaS towards increasing the productivity of humans and systems, and in avoiding wrong decisions involving mission-critical systems.