CNS Core: SMALL: Clarifying Experimenter Bias by Identifying and Visualizing Experiment Bottlenecks

CNS 核心：SMALL：通过识别和可视化实验瓶颈来澄清实验者偏见

• 批准号: 1908020
• 负责人: Yang Wang
• 金额: $ 49.69万
• 依托单位: Ohio State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2023-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908020&HistoricalAwards=false
• 关键词: CNS; Core; SMALL; Clarifying; Experimenter

Running experiments to measure the performance of a system and comparing its performance to other works is a routine procedure of computer science research. However, an experimenter usually has a natural tendency to choose the experimental setting in favor of his/her own work. This may introduce unfairness into the comparison. Instead of trying to eliminate such bias, which is difficult because human effort is unavoidable, this project tries to clarify such bias to readers by visualizing the experimental results so that a reader can understand the true intention of the experimenter.To achieve this goal, this project observes that an experimenter usually introduces bias by tuning the bottleneck of his/her experiment. This project targets developing comprehensive, automatic, and efficient approaches to identify experiment bottlenecks and concise and informative approaches to present such bottlenecks to readers. To be concrete, this project proposes to use a wait-for graph and an interpretive cumulative distribution function (CDF) to identify and present bottlenecks related to throughput and latency respectively, which are the most widely used performance metrics. It will further investigate how to scale these mechanisms to large and long-running systems.If successful, this project has the potential for significant real-world impact, because first, performance analysis is a routine procedure in both industry and academia, and second, clarifying experimenter bias would allow our community to better evaluate a research work. This project will further provide materials to educate students in a variety of programs, including undergraduate and graduate operating systems courses in the Department of Computer Science and Engineering, new projects in the Department of Analytics, and new projects at the Ohio Hackathon (targeting undergraduate and high school students).All versions of data, including program source code, papers, trace data, and other documentation will be maintained in a data repository at the Ohio State University using a versioning infrastructure, with periodic back up. The update of all data will be published at http://web.cse.ohio-state.edu/~yangwang/experimenter_bias.html. All data will be stored and maintained for at least five years after the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

运行实验以衡量系统的性能并将其性能与其他作品进行比较是计算机科学研究的常规过程。但是，实验者通常有自然的倾向，可以选择实验环境以支持自己的工作。 这可能会将不公平引入比较中。该项目并没有试图消除这种偏见，这很困难，因为人类的努力是不可避免的，而是试图通过可视化实验结果来澄清读者的这种偏见，以便读者可以理解实验者的真实意图。要实现这一目标，该项目观察到，一个实验者通常通过调谐他的/她的实验来引入偏见。该项目针对开发全面，自动和高效的方法来识别实验瓶颈以及简洁而有益的方法，以向读者展示此类瓶颈。为了具体，该项目建议使用等待图和解释性累积分布函数（CDF）分别识别和呈现与吞吐量和潜伏期相关的瓶颈，这是最广泛使用的性能指标。 它将进一步研究如何将这些机制扩展到大型且长期运行的系统。如果成功，该项目有可能产生重大的现实影响，因为首先，绩效分析是行业和学术界的常规程序，其次，澄清实验者的偏见将使我们的社区能够更好地评估研究工作。该项目将进一步提供材料，以教育各种课程，包括计算机科学和工程系的本科和研究生操作系统课程，分析部的新项目以及俄亥俄州黑客马拉松的新项目（针对本科生和高中生）（所有的数据源），包括计划的数据，包括计划源代码，涉及数据，以及其他文档，以及其他文档，以及其他文档，以及其他范围的数据，并在其他文档中进行了介绍。基础架构，并定期备份。所有数据的更新将在http://web.cse.ohio-state.edu/~yangwang/experimenter_bias.html上发布。该项目完成后，所有数据将至少存储和维护五年。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响标准，被认为值得通过评估来获得支持。

项目成果

IsoBugView: Interactively Debugging Isolation Bugs in Database Applications

• DOI: 10.14778/3554821.3554885
• 发表时间: 2022-08
• 期刊: Proc. VLDB Endow.
• 作者: Drew Ripberger;Yifan Gan;Xueyuan Ren;Spyros Blanas;Yang Wang

Finding heterogeneous-unsafe configuration parameters in cloud systems

• DOI: 10.1145/3447786.3456250
• 发表时间: 2021-04
• 期刊: Proceedings of the Sixteenth European Conference on Computer Systems
• 作者: Sixiang Ma;Fang Zhou;Michael D. Bond;Yang Wang

SICS: Secure and Dynamic Middlebox Outsourcing

• DOI: 10.1109/tnet.2020.3023386
• 发表时间: 2020-12
• 期刊: IEEE/ACM Transactions on Networking
• 作者: Huazhe Wang;Xin Li;Chen Qian

On the Discontinuation of Persistent Memory: Looking Back to Look Forward

论持久内存的终止：回顾过去展望未来

• 发表时间: 2023
• 期刊: USA Co-located with ISCA 2023
• 作者: Li, Tianxi;Wang, Yang;Lu, Xiaoyi
• 通讯作者: Lu, Xiaoyi