Three pitfalls in Java performance evaluation
Three pitfalls in Java performance evaluation
复制标题
Java性能评估的三个陷阱
DOI：
--
复制
发表时间：
2008
期刊：
影响因子：
--
通讯作者：
A. Georges
中科院分区：
文献类型：
--
作者：
A. Georges
Java编程语言在过去十年中有了显著的发展。这部分是由于在通用微处理器上运行Java应用程序所需的基础设施：Java虚拟机（VM）。VM确保Java应用程序在不同硬件平台之间是可移植的，因为它将应用程序与底层系统隔离开来。因此有了“写一次，跑（几乎）到任何地方”的格言。Java应用程序被编译成一种中间形式，称为字节码，它由许多所谓的类文件组成。虚拟机负责类加载、将字节码解释或编译为底层硬件平台的本机代码、线程调度、垃圾收集等。因此，在Java应用程序执行期间，VM会定期进行干预，以处理内务管理任务，并在执行时优化应用程序。此外，大多数虚拟机的具体实现细节将不确定性行为插入到执行的语义部分，而不是插入到较低级别的执行中。例如，要使Java应用程序达到与用C等语言编写的经典编译程序竞争的速度，虚拟机需要优化Java字节码。为了限制执行开销，大多数虚拟机使用时间采样机制来确定应用程序中的热门方法。这引入了不确定性，因为在几次运行中，方法并不总是在同一时刻优化，优化的方法集也不总是相同的。其他引入不确定性的因素包括线程调度、垃圾收集等。很容易看出，Java应用程序的性能分析并不像最初看起来那么简单，需要更仔细地检查。在本论文中，我们主要对Java应用程序的行为及其性能感兴趣。在这项工作的过程中，我们发现了研究人员在分析Java性能之前没有考虑到的三个主要缺陷。我们将简要总结本文的主要成果。我们提出的第一个缺陷涉及虚拟机、应用程序和应用程序输入之间的交互。短期运行应用程序的性能主要由虚拟机决定。对于长时间运行的应用程序，这种影响会减小，但仍然是明显的。我们使用统计分析，如主成分分析和聚类分析（K-means和分层聚类）来证明和澄清陷阱。通过使用硬件性能计数器、5个虚拟机和14个具有大小输入大小的基准测试来测量大量性能特征，我们证明了短时间运行的工作负载主要由虚拟机集群。即使对于来自SPECjvm98基准套件的长时间运行的应用程序，虚拟机仍然对微体系结构级别上观察到的行为产生很大的影响。这项工作表明，需要比以前更大、运行时间更长的基准测试——DaCapo基准测试套件的引入（部分地）满足了这一点——并且在设置实验以避免测量虚拟机而不是基准测试时需要仔细考虑。在此工作之前，人们经常对短时间运行的应用程序使用模拟（以节省时间）来探索Java性能。我们发现的第二个陷阱涉及对性能数字的分析。在过去七年中，在OOPSLA、PLDI、CGO、ISMM和VEE等主要会议上发表的50篇论文中，我们发现使用了各种方法，既用于实验设计（例如，输入大小、虚拟机、堆大小等），更重要的是，用于数据分析（例如，使用三分之一的最佳性能数字）。新技术与使用这些流行方法的现有工作相抗衡，关于它们是否成功击败先前的最先进技术的结论是基于它们的。考虑到Java应用程序的执行通常涉及虚拟机中的不确定性（例如，在确定要优化哪些方法时），这些流行方法中缺乏统计严密性会导致误导性甚至错误的结论也就不足为奇了。我们的意思是，这些结论要么不能代表实际发生的情况，要么甚至与以统计方式建模的现实相矛盾。为了规避这个陷阱，我们提出了一种严格的统计方法，使用置信区间来报告和比较性能数字。我们还声称，应该进行充分的实验，以获得可靠的性能测量。由虚拟机中基于计时器的优化组件引起的不确定性可以使用所谓的重放编译来消除。该技术将在应用程序的首次执行或分析运行期间记录编译计划。在第二次执行期间，应用程序迭代两次：一次编译并优化编译计划中找到的所有方法，第二次执行实际测量。然而，事实证明，当前使用单个计划（对应于性能最好的分析运行）或选择在半数以上的分析运行中得到优化的方法的组合计划的实践，都无法与使用多个计划相匹配。在计划本身中观察到的可变性太大，无法在当前的实践中捕获。因此，使用多个计划绝对是更好的选择。此外，这允许在数据分析中使用匹配对方法，从而为平均性能数提供更紧凑的置信区间。我们研究的第三个缺陷是在调优应用程序或虚拟机时使用全局性能数字。我们展示了Java应用程序在方法级别展示了阶段行为。这意味着同一方法的实例在行为上比其他方法的实例表现出更多的相似性。然后可以将阶段识别为动态调用树的一组子树，每个子树都以相同的方法为首。我们提出了一种两步算法，允许将步骤2中的硬件性能计数器数据与步骤1中确定的阶段相关联。所获得的信息可以用于向程序员显示哪些方法的性能低于平均水平，例如，它们所导致的缓存丢失次数。在本文中，我们特别注意统计的严谨性。对于每个陷阱，我们使用统计数据来证明它的存在。希望这项工作能鼓励其他研究人员在他们的工作中也使用更严格的方法。
The Java programming language has known a remarkable growth over the last decade. This is partially due to the infrastructure required to run Java ap- plications on general purpose microprocessors: a Java virtual machine (VM). The VM ensures that Java applications are portable across different hardware platforms, because it shelters the applications from the underlying system. Hence the motto write once, run (almost) anywhere. Java applications are compiled to an intermediate form, called bytecode, and consist of a number of so-called class files. The virtual machine takes care of class loading, interpreting or compiling the bytecode to the native code of the underlying hardware platform, thread scheduling, garbage collection, etc. As such, during the execution of a Java application, the VM regularly intervenes to take care of housekeeping tasks and to optimise the application as it is executing. Furthermore, the specific implementation details of most virtual machines insert non-deterministic behaviour, not into the semantic part of the execution, but rather into the lower level execution. For example, to bring a Java application up to competitive speed with classical compiled programs written in languages such as C, the virtual machine needs to optimise Java bytecode. To limit the execution overhead, most virtual machines use a time sampling mechanism to determine the hot methods in the application. This introduces non-determinism, as over several runs, the methods are not always optimised at the same moment, nor is the set of optimised methods always the same. Other factors that introduce non-determinism are the thread scheduling, garbage collection, etc. It is readily seen that performance analysis of Java applications is not as simple as it seems at first, and warrants closer inspection. In this dissertation we are mainly interested in the behaviour of Java applications and their performance. In the course of this work, we uncovered three major pitfalls that were not taken into account by researchers when analysing Java performance prior to this work. We will briefly summarise the main achievements presented in this dissertation. The first pitfall we present involves the interaction between the virtual machine, the application and the input to the application. The performance for short running applications is shown to be mainly determined by the virtual machine. For longer running applications, this influence decreases, but remains tangible. We use statistical analysis, such as principal components analysis and cluster analysis (K-means and hierarchical clustering) to demonstrate and clarify the pitfall. By means of a large number of performance char- acteristics measured using hardware performance counters, five virtual machines and fourteen benchmarks with both a small and a large input size, we demonstrate that short running workloads are primarily clustered by virtual machines. Even for long running applications from the SPECjvm98 benchmark suite, the virtual machine still exerts a large influence on the observed behaviour at the microarchitectural level. This work has shown the need for both larger and longer running benchmarks than were available prior to it – this was (partially) met by the introduction of the DaCapo benchmark suite – as well as a careful consideration when setting up an experiment to avoid measuring the virtual machine, rather than the benchmark. Prior to this work, people were quite often using simulation with short running applications (to save time) for exploring Java performance. The second pitfall we uncover involves the analysis of performance numbers. During a survey of 50 papers published at premier conferences, such as OOPSLA, PLDI, CGO, ISMM and VEE, over the past seven years, we found that a variety of approaches are used, both for experimental design – for example, the input size, virtual machines, heap sizes, etc. – and, even more importantly, for data analysis – for example, using a best out of 3 performance number. New techniques are pitted against existing work using these prevalent approaches, and conclusions regarding their successfulness in beating prior state-of-the-art are based upon them. Given the fact that the execution of Java applications usually involves non-determinism in the virtual machine – for example, when determining which methods to optimise – it should come as no surprise that the lack of statistical rigour in these prevalent approaches leads to misleading or even incorrect conclusions. By this we mean that the conclusions are either not representative of what actually happens, or even contradict reality, as modelled in a statistical manner. To circumvent this pitfall, we propose a rigorous statistical approach that uses confidence intervals to both report and compare performance numbers. We also claim that sufficient experiments should be conducted to get a reliable performance measure. The non-determinism caused by the timer-based optimisation component in a virtual machine can be eliminated using so-called replay compilation. This technique will record a compilation plan during a first execution or profiling run of the application. During a second execution, the application is iterated twice: once to compile and optimise all methods found in the compilation plan, and a second time to perform the actual measurement. It turns out however that current practice of using either a single plan – corresponding to the best performing profiling run – or a combined plan choosing the methods that were optimised in, say, more than half the profiling runs, is no match for using multiple plans. The variability observed in the plans themselves is too large to capture in one of the current practices. Consequently, using multiple plans is definitely the better option. Moreover, this allows using a matched-pair approach in the data analysis, which results in tighter confidence intervals for the mean performance number. The third pitfall we examine is the usage of global performance numbers when tuning either an application or a virtual machine. We show that Java applications exhibit phase behaviour at the method level. This means that instances of the same method show more similarity to each other, behaviourwise, than to instances of other methods. A phase can then be identified as a set of sub-trees of the dynamic call-tree, with each sub-tree headed by the same method. We present an two-step algorithm that allows correlating hardware performance counter data in step 2 with the phases determined in step 1. The information obtained can be applied to show the programmer which methods perform worse than average, for example with respect to the number of cache misses they incur. In the dissertation, we pay particular attention to statistical rigour. For each pitfall, we use statistics to demonstrate its presence. Hopefully this work will encourage other researchers to use more rigour in their work as well.