Rigorous Empirical Performance Analysis of Embedded Systems

点击翻译按钮获取中文摘要

Empirical systems research is facing a dilemma: getting accurate measurements is difficult and time consuming. Minor aspects of an experimental setup can have a significant impact on performance measurements and potentially invalidate conclusions drawn. The growth in complexity and size of modern systems will further exacerbate this dilemma. So how can one trust any reported empirical analysis of a new idea or concept in computer science? The long-term objective of the applicant's research is to understand how to create reproducible, robust, and reliable performance experiments with reasonable effort for the embedded domain. This work is important because, as computer science matures, even more subareas will report lower improvement rates and hence our evaluation techniques must increase in accuracy to distinguish real innovation from flukes. Towards this long-term goal, we propose a program to develop a systematic approach to identify hidden factors influencing performance analysis across the system stack, investigate their interaction, and study transferability of results, and integrate the results into our experimentation infrastructure. This program targets the domain of embedded software, because embedded systems are usually resource constrained (e.g., low memory, low processing power, battery driven) and thus have a particular need for performance analysis. However, the results such as newly found factors and the tools will also be applicable to general systems. The program will have significant impact through its practical results and also the developed tools. Our research targets a critically needed technology (i.e., enabling accurate performance measurements). Canada's Information and Communiation Technology (ICT) needs this to make decisions on sound data.