Rigorous Empirical Performance Analysis of Embedded Systems

经验系统研究正面临两难境地：获得准确的测量既困难又耗时。实验设置的次要方面可能会对性能测量产生重大影响，并可能使得出的结论无效。现代系统的复杂性和规模的增长将进一步加剧这一困境。那么，人们怎么能相信对计算机科学中新思想或新概念的任何报道的经验分析呢？ 申请人研究的长期目标是了解如何以合理的努力为嵌入式领域创建可重复性、健壮性和可靠性的性能实验。这项工作很重要，因为随着计算机科学的成熟，更多的子领域将报告更低的改善率，因此我们的评估技术必须提高准确性，以区分真正的创新和侥幸。 为了实现这一长期目标，我们提出了一项计划，以开发一种系统的方法来识别影响整个系统堆栈的性能分析的隐藏因素，调查它们的交互作用，研究结果的可转移性，并将结果集成到我们的实验基础设施中。该程序针对嵌入式软件领域，因为嵌入式系统通常受资源限制(例如，低内存、低处理能力、电池驱动)，因此特别需要进行性能分析。然而，新发现的因子和工具等结果也将适用于一般系统。 该方案将通过其实际效果和开发的工具产生重大影响。我们的研究目标是一种迫切需要的技术(即，能够进行准确的性能测量)。加拿大的信息和通信技术(ICT)需要这一点来对健全的数据做出决定。

Empirical systems research is facing a dilemma: getting accurate measurements is difficult and time consuming. Minor aspects of an experimental setup can have a significant impact on performance measurements and potentially invalidate conclusions drawn. The growth in complexity and size of modern systems will further exacerbate this dilemma. So how can one trust any reported empirical analysis of a new idea or concept in computer science? The long-term objective of the applicant's research is to understand how to create reproducible, robust, and reliable performance experiments with reasonable effort for the embedded domain. This work is important because, as computer science matures, even more subareas will report lower improvement rates and hence our evaluation techniques must increase in accuracy to distinguish real innovation from flukes. Towards this long-term goal, we propose a program to develop a systematic approach to identify hidden factors influencing performance analysis across the system stack, investigate their interaction, and study transferability of results, and integrate the results into our experimentation infrastructure. This program targets the domain of embedded software, because embedded systems are usually resource constrained (e.g., low memory, low processing power, battery driven) and thus have a particular need for performance analysis. However, the results such as newly found factors and the tools will also be applicable to general systems. The program will have significant impact through its practical results and also the developed tools. Our research targets a critically needed technology (i.e., enabling accurate performance measurements). Canada's Information and Communiation Technology (ICT) needs this to make decisions on sound data.