Rigorous Empirical Performance Analysis of Embedded Systems

经验系统研究正面临着一个困境：获得准确的测量是困难和耗时的。实验设置的次要方面可能会对性能测量产生重大影响，并可能使得出的结论无效。现代系统的复杂性和规模的增长将进一步加剧这一困境。那么，人们如何能够相信任何对计算机科学中新思想或新概念的经验分析呢？ 申请人研究的长期目标是了解如何在嵌入式领域通过合理的努力创建可重现的，健壮的和可靠的性能实验。这项工作很重要，因为随着计算机科学的成熟，更多的子领域将报告较低的改进率，因此我们的评估技术必须提高准确性，以区分真实的创新侥幸。 为了实现这一长期目标，我们提出了一个计划，以开发一个系统的方法来识别隐藏的因素影响整个系统堆栈的性能分析，调查他们的相互作用，研究结果的可转移性，并将结果整合到我们的实验基础设施。该计划的目标是嵌入式软件领域，因为嵌入式系统通常是资源受限的（例如，低存储器、低处理能力、电池驱动），因此特别需要性能分析。然而，新发现的因素和工具等结果也适用于一般系统。 该计划将通过其实际结果和开发的工具产生重大影响。我们的研究目标是一项急需的技术（即，能够实现精确的性能测量）。加拿大的信息和通信技术（ICT）需要这一点来做出关于可靠数据的决定。

Empirical systems research is facing a dilemma: getting accurate measurements is difficult and time consuming. Minor aspects of an experimental setup can have a significant impact on performance measurements and potentially invalidate conclusions drawn. The growth in complexity and size of modern systems will further exacerbate this dilemma. So how can one trust any reported empirical analysis of a new idea or concept in computer science? The long-term objective of the applicant's research is to understand how to create reproducible, robust, and reliable performance experiments with reasonable effort for the embedded domain. This work is important because, as computer science matures, even more subareas will report lower improvement rates and hence our evaluation techniques must increase in accuracy to distinguish real innovation from flukes. Towards this long-term goal, we propose a program to develop a systematic approach to identify hidden factors influencing performance analysis across the system stack, investigate their interaction, and study transferability of results, and integrate the results into our experimentation infrastructure. This program targets the domain of embedded software, because embedded systems are usually resource constrained (e.g., low memory, low processing power, battery driven) and thus have a particular need for performance analysis. However, the results such as newly found factors and the tools will also be applicable to general systems. The program will have significant impact through its practical results and also the developed tools. Our research targets a critically needed technology (i.e., enabling accurate performance measurements). Canada's Information and Communiation Technology (ICT) needs this to make decisions on sound data.