SI2-SSE: EASE: Improving Research Accountability through Artifact Evaluation

SI2-SSE：EASE：通过工件评估提高研究问责制

• 批准号: 1535232
• 负责人: Bruce Childers
• 金额: $ 49.95万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1535232&HistoricalAwards=false
• 关键词: SI2; SSE; EASE; Improving; Research

Research in computer systems, particularly in the first stages of creating a new innovation, relies almost exclusively on software prototypes, simulators, benchmarks, and data sets to understand the benefits and costs of new ideas in computers, ranging from consumer devices to exascale systems. These artifacts are used to evaluate new capabilities, algorithms, bottlenecks and trade-offs. Empirical study is behind the rapid pace of innovation in creating faster, lower energy and more reliable systems. This experimental approach lies at the core of development that fuels the nation's information economy. Given the critical importance of experimental study to developing new computer systems, several efforts are underway to curate experimental results through accountable research. One effort, Artifact Evaluation (AE), is being adopted to promote high quality artifacts and experimentation, including making public the experimental information necessary for reproducibility. However, the rapid adoption of AE is hampered by technical challenges that create a high barrier to the process: there is no consistent or simple environment, or mechanism, to package and reproduce experiments for AE. Authors rely on their own approaches, leading to much time consumed, as well as considerable variability in the ways materials are prepared and evaluated, unnecessarily obstructing the AE process. To overcome the technical challenges with AE, and to more broadly encourage adoption of AE in computer science and engineering research, this project is developing a software infrastructure, Experiment and Artifact System for Evaluation (EASE), to create and run experiments specifically for AE, in which authors create, conduct and share artifacts and experiments. It allows for repeating, modifying, and extending experiments. Authors may also use EASE to package and upload their experiments for archival storage in a digital library. EASE is being developed and deployed for two use cases, namely compilers and real-time systems, keeping the project tractable to address specific needs. These communities have overlapping but also distinct requirements, helping to ensure EASE can also be extended andused by other computer systems research communities as well.EASE will be release as open source software, based on an Experiment Management System (EMS) previously developed by the project investigator in a project call Open Curation for Computer Architecture Modeling (OCCAM), used to define and conduct experiments using computer architecture simulators. Using EMS as a starting point, EASE will provide AE support, by: 1) separating EMS from OCCAM's repository and hardware services, transforming the EMS infrastructure into EASE, a fully standalone, sustainable, and extensible platform for AE; 2) supporting record and replay (for repeating and reproducing results, as well as provenance) of artifacts and experiments as part of normal development and experimental practice to ease participation in AE by authors and evaluators; 3) supporting artifacts,workflows of artifacts and experiments that run directly on a machine, including specialized hardware and software, and run indirectly on a simulator or emulator; 4) allowing both user-level (artifacts and experiments as user processes) and system-level (artifacts and experiments involving kernel changes) innovations; 5) providing consistent/uniform access, whether locally or remotely, to artifacts and experiments; 6) simplifying viewing, running, modifying, and comparing experiments by innovators (i.e., during innovation development), artifact evaluators (during AE), and archive users (after publication); 7) enabling indexing (object locators and search tags) and packaging of artifacts and experiments for AE and for archival deployment (e.g., to ACM?s or IEEE?s Digital Library); and 8) refining, expanding, generalizing, and documenting EASE to ensure it is robust, maintainable and extensible, and that it can be used and sustained by different CSR communities (starting with real-time and compilers, given their different artifacts, data and methods).

计算机系统的研究，特别是在创造新创新的第一阶段，几乎完全依赖软件原型、模拟器、基准和数据集来了解计算机中新想法的好处和成本，从消费设备到亿级系统。这些构件用于评估新功能、算法、瓶颈和权衡。在创造更快、更低能耗和更可靠系统方面，快速创新步伐的背后是经验研究。这种实验性的方法是推动国家信息经济发展的核心。鉴于实验研究对开发新的计算机系统的关键重要性，目前正在进行几项努力，以通过负责任的研究来管理实验结果。一项名为人工制品评估(AE)的努力正在被采用，以促进高质量的人工制品和实验，包括公开重复性所需的实验信息。然而，声发射的快速采用受到技术挑战的阻碍，这些挑战对这一过程造成了很高的障碍：没有一致或简单的环境或机制来包装和复制声发射实验。作者依靠自己的方法，导致耗费大量时间，以及材料准备和评估方式的相当大的差异，不必要地阻碍了声发射过程。为了克服声发射的技术挑战，并更广泛地鼓励在计算机科学和工程研究中采用声发射，该项目正在开发一个软件基础设施、实验和评估人工制品系统(EASE)，以创建和运行专门针对声发射的实验，在其中作者创建、进行和共享人工制品和实验。它允许重复、修改和扩展实验。作者还可以使用Easy打包并上传他们的实验，以便在数字图书馆中存档存储。正在为两个用例开发和部署Easy，即编译器和实时系统，使项目易于处理，以满足特定需求。这些社区有重叠但也有不同的要求，有助于确保EASE也可以被其他计算机系统研究社区扩展和使用。EASE将作为开源软件发布，基于项目调查员之前在一个名为Open Cation for Computer Architecture Model(OCCAM)的项目中开发的实验管理系统(EMS)，该系统用于定义和进行使用计算机体系结构模拟器的实验。以EMS为起点，EASE将通过以下方式提供AE支持：1)将EMS与Occam的存储库和硬件服务分开，将EMS基础设施转变为EASY，一个完全独立、可持续和可扩展的AE平台；2)支持人工制品和实验的记录和回放(用于重复和再现结果，以及来源)，作为正常开发和实验实践的一部分，以方便作者和评估者参与AE；3)支持直接在机器上运行的人工产物、人工产物的工作流和实验，包括专门的硬件和软件，并间接运行在模拟器或仿真器上；4)允许用户级(作为用户进程的人工产物和实验)和系统级(涉及内核变化的人工产物和实验)创新；5)提供对人工产物和实验的一致/统一访问，无论是本地还是远程；6)简化创新者(即在创新开发期间)、人工产物评估者(在AE期间)和档案用户(发布后)对实验的查看、运行、修改和比较；7)启用索引(对象定位器和搜索标签)，并为AE和档案部署(例如，到ACM？S或IEEE？S数字图书馆)对人工产物和实验进行打包和打包；8)提炼、扩展、概括和记录Easy，以确保它是健壮的、可维护的和可扩展的，并确保它可以由不同的CSR社区使用和维护(从实时和编译器开始，因为它们的构件、数据和方法不同)。