CNS Core: SMALL: Clarifying Experimenter Bias by Identifying and Visualizing Experiment Bottlenecks

运行实验来测量系统的性能并将其性能与其他作品进行比较是计算机科学研究的常规过程。然而，实验者通常有一种自然的倾向，选择有利于他/她自己工作的实验环境。 这可能会在比较中引入不公平。本项目没有试图消除这种偏见，因为这是困难的，因为人类的努力是不可避免的，而试图通过可视化的实验结果向读者澄清这种偏见，使读者能够理解实验者的真实意图。为了实现这一目标，本项目观察到，实验者通常通过调整他/她的实验瓶颈来引入偏见。该项目的目标是开发全面，自动和有效的方法来识别实验瓶颈和简洁和翔实的方法来呈现这些瓶颈给读者。具体来说，该项目提出使用等待图和解释性累积分布函数（CDF）来识别和呈现与吞吐量和延迟相关的瓶颈，这是最广泛使用的性能指标。 它将进一步研究如何将这些机制扩展到大型和长期运行的系统。如果成功，这个项目有可能产生重大的现实影响，因为首先，性能分析是工业界和学术界的常规程序，其次，澄清实验者的偏见将使我们的社区能够更好地评估研究工作。该项目将进一步提供材料，以教育学生在各种方案，包括本科和研究生操作系统课程在计算机科学与工程系，在分析部门的新项目，并在俄亥俄州黑客学院的新项目（针对本科生和高中生）。所有版本的数据，包括程序源代码，论文，跟踪数据，和其他文件将使用版本控制基础设施保存在俄亥俄州州立大学的数据存储库中，并定期备份。所有数据的更新将在http://web.cse.ohio-state.edu/~yangwang/experimenter_bias.html上发布。所有数据将在项目完成后保存和维护至少五年。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Running experiments to measure the performance of a system and comparing its performance to other works is a routine procedure of computer science research. However, an experimenter usually has a natural tendency to choose the experimental setting in favor of his/her own work. This may introduce unfairness into the comparison. Instead of trying to eliminate such bias, which is difficult because human effort is unavoidable, this project tries to clarify such bias to readers by visualizing the experimental results so that a reader can understand the true intention of the experimenter.To achieve this goal, this project observes that an experimenter usually introduces bias by tuning the bottleneck of his/her experiment. This project targets developing comprehensive, automatic, and efficient approaches to identify experiment bottlenecks and concise and informative approaches to present such bottlenecks to readers. To be concrete, this project proposes to use a wait-for graph and an interpretive cumulative distribution function (CDF) to identify and present bottlenecks related to throughput and latency respectively, which are the most widely used performance metrics. It will further investigate how to scale these mechanisms to large and long-running systems.If successful, this project has the potential for significant real-world impact, because first, performance analysis is a routine procedure in both industry and academia, and second, clarifying experimenter bias would allow our community to better evaluate a research work. This project will further provide materials to educate students in a variety of programs, including undergraduate and graduate operating systems courses in the Department of Computer Science and Engineering, new projects in the Department of Analytics, and new projects at the Ohio Hackathon (targeting undergraduate and high school students).All versions of data, including program source code, papers, trace data, and other documentation will be maintained in a data repository at the Ohio State University using a versioning infrastructure, with periodic back up. The update of all data will be published at http://web.cse.ohio-state.edu/~yangwang/experimenter_bias.html. All data will be stored and maintained for at least five years after the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.