CSR: Small: BaaS: Benchmarking-As-A-Service for Improved Visibility, Diagnosis, and Control of Modern Data Analytics Platforms

CSR：小型：BaaS：基准测试即服务，以提高现代数据分析平台的可见性、诊断和控制

• 批准号: 1423128
• 负责人: Shivnath Babu
• 金额: $ 45万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1423128&HistoricalAwards=false
• 关键词: CSR; Small; BaaS; Benchmarking; Service

System benchmarking is a critical technique used by application developers and system administrators to help them decide how best to perform a wide spectrum of system management tasks. These tasks range from design-time aspects such as picking which system to use for an application, to all pre-deployment aspects of testing and tuning the applications and system, and to production aspects such as reliability assurance and problem diagnosis. However, the reality today is that the massive potential of benchmarking is seldom harnessed satisfactorily by application developers and system administrators. Benchmarking is very human-effort-intensive. Users may not be aware of or follow correct benchmarking methodology. Often corners are cut to reduce either time or cost without full awareness of the implications of doing so. Repeatability and reuse of benchmarking results are often unattainable. The results of a benchmarking effort may be misinterpreted, leading to incorrect or suboptimal decisions.This project combines new research with recent technology trends in order to make the power of benchmarking accessible to a diverse community of users. The project is building a Benchmarking-as-a-Service (BaaS) platform that runs on top of an Infrastructure-as-a-Service or Platform-as-a-Service cloud platform. A BaaS provides four integrated services to simplify the overall benchmarking process, while addressing a number of novel technical challenges: 1. Design: Helps the user create the workloads and data that best meet her specific benchmarking needs, using the concept of benchmark design as a search problem; 2. Execute: Helps the user run the selected workloads on appropriate data, system configurations, and hardware and software resources in a timely and cost-efficient manner; 3. Interpret: Helps the user interpret the results of the benchmark correctly since many users who run benchmarks on systems are far from being experts on these systems (many challenges arise here due to the large volume, variety, and velocity of system monitoring data); and 4. Build: Helps the user build new system management services on top of the BaaS. For example, BaaS enables the automatic tuning of production database instances running on cloud platforms like Amazon Web Services and SQL Azure. BaaS is being prototyped and opportunities for deploying it in real-world settings will be pursued. The prototype will be used to evaluate the transformative capabilities of BaaS towards increasing the productivity of humans and systems, and in avoiding wrong decisions involving mission-critical systems.

系统基准测试是应用程序开发人员和系统管理员用来帮助他们决定如何最好地执行广泛的系统管理任务的关键技术。这些任务的范围从设计时方面（如选择应用程序使用哪个系统）到测试和调整应用程序和系统的所有部署前方面，以及生产方面（如可靠性保证和问题诊断）。然而，今天的现实是，应用程序开发人员和系统管理员很少满意地利用基准测试的巨大潜力。基准测试是非常耗费人力的。用户可能没有意识到或遵循正确的基准测试方法。 为了减少时间或成本，往往会偷工减料，而没有充分意识到这样做的后果。基准测试结果的可重复性和重复使用性往往无法实现。基准测试工作的结果可能会被误解，导致不正确或次优的决策。该项目将新的研究与最新的技术趋势相结合，以使不同社区的用户可以使用基准测试的力量。该项目正在构建一个基准即服务（BaaS）平台，该平台运行在云结构即服务或平台即服务云平台之上。BaaS提供了四个集成的服务来简化整体基准测试过程，同时解决了许多新的技术挑战：1.设计图：帮助用户创建最能满足其特定基准测试需求的工作负载和数据，使用基准测试设计的概念作为搜索问题; 2.执行：帮助用户以及时且经济高效的方式在适当的数据、系统配置以及硬件和软件资源上运行选定的工作负载; 3.翻译：帮助用户正确解释基准测试的结果，因为许多在系统上运行基准测试的用户远非这些系统的专家（由于系统监控数据的大量，多样性和速度，这里出现了许多挑战）;以及4.构建：帮助用户在BaaS之上构建新的系统管理服务。例如，BaaS支持自动调优在Amazon Web Services和SQL Azure等云平台上运行的生产数据库实例。BaaS正在原型化，并将寻求在现实世界中部署它的机会。该原型将用于评估BaaS的变革能力，以提高人类和系统的生产力，并避免涉及关键任务系统的错误决策。