CaMELot: Catching and Mitigating Event-Loop Concurrency Issues

大多数现代计算机应用程序都以某种方式依赖于由互联网上的服务器应用程序执行的计算。越来越多的服务器应用程序现在被构建为所谓的微服务，允许开发人员逐步更新或修复较大应用程序中不相关部分的问题，因此变得流行。许多微服务通过设计避免了某些类型的并发问题。不幸的是，他们仍然受到其他类型的并发问题的困扰，例如，当多个在线客户试图同时预订相同的座位时。对于软件工程师来说，很难测试所有可能的并发交互。在实践中，这意味着在测试过程中只能可靠地检测到简单的并发问题。然而，复杂的问题很容易通过并进入服务器应用程序，然后错误地处理客户端请求。当Facebook股票首次交易时，纳斯达克出现了这样一个并发问题的例子，导致数百万美元的损失。我们的目标是开发在运行时自动检测并发问题的技术，能够绕过它们，并使开发人员能够使用检测技术收集的详细信息来修复它们。研究人员已经证明，人们可以检测和避免问题，例如通过改变客户端请求的处理顺序。然而，在实践中，当前的技术使服务器应用显著地减慢，这使得这些技术太昂贵而不能使用。我们的目标是动态地平衡对准确信息的需求，并最大限度地降低速度。我们推测，我们可以获得最实际的好处，同时很少跟踪程序代码执行方式的精确细节。除了自动防止并发问题导致问题之外，我们还将使用获得的信息向开发人员提供反馈，以便他们可以修复软件中的底层问题。因此，本研究项目的总体目标是使服务器应用程序，特别是微服务，对于难以测试的软件错误，我们的工作将更加强大和灵活，因此通常不会被发现，直到它们给客户或公司造成重大问题。导致开发了自适应技术，这些技术可以检测并发问题，并自动权衡准确性和运行时开销，以便在实践中使用。此外，检测技术将用于为软件开发人员提供可操作的输入，以便可以修复并发问题，从而在未来可靠地防止并发问题。为了评估这项工作，我们将收集各种不同类型的并发问题，并将其公开提供。该系列将基于工业系统的问题，并从高度复杂的错误的理论场景中得出。我们包括这些理论的情况下，因为这样复杂的错误是很难诊断和测试，他们可能仍然未被诊断和记录在实践中，但有可能造成重大disruption.Finally，我们将建立和评估我们提出的技术的基础上设计的并发研究系统。该系统使用Oracle实验室的GraalVM技术，这使我们能够在最先进的系统水平上进行原型设计，同时保持开发工作可由小型团队管理。

Most modern computer applications depend in some way or another on computations that are performed by server applications on the internet. More and more of these server applications are now built as so-called microservices, which allow developers to gradually update or fix issues in unrelated parts of a larger application, and therefore, have become popular. Many of these microservices avoid certain types of concurrency issues by design. Unfortunately, they still suffer from other kinds of concurrency issues, for example when multiple online customers try to reserve the same seats at the same time.For software engineers, it is hard to test for all possible concurrent interactions. In practice, this means that only simple concurrency issues are reliably detected during testing. Complex issues can however easily slip through and make it into server applications and then handle client requests incorrectly. One example of such a concurrency issue appeared at Nasdaq when the Facebook stock was traded for the first time, resulting in the loss of millions of dollars.Our goal is to develop techniques that detect concurrency issues automatically at run time, to be able to circumvent them, and enable developers to fix them, using detailed information gathered by the detection techniques. Researchers have shown that one can detect and avoid issues, for instance by changing the order in which client requests are processed. In practice however, current techniques slow server applications down significantly, which make these techniques too costly to be used. Our aim is to dynamically balance the need for accurate information and minimize slow down. We conjecture that we can get most practical benefits while only rarely tracking precise details of how program code executes. In addition to automatically preventing concurrency issues to cause problems, we will also use the obtained information to provide feedback to developers so that they can fix the underlying issue in their software.Thus, overall the goal of this research project is to make server applications, and specifically microservices, more robust and resilient to software bugs that are hard to test for and therefore typically remain undiscovered until they cause major issues for customers or companies.Our work will result in the development of adaptive techniques that detect concurrency issues, and automatically tradeoff accuracy and run-time overhead, to be usable in practice. Furthermore, the detection techniques will be used to provide actionable input to the software developers, so that the concurrency issue can be fixed and therefore be prevented reliably in the future.To evaluate this work, we will collect various different types of concurrency issues and make them openly available. This collection will be based on issues from industrial systems and derived from theoretical scenarios for highly complex bugs. We include these theoretical scenarios, since such complex bugs are hard to diagnose and test for, they likely remain undiagnosed and undocumented in practice, but have the potential of causing major disruptions.Finally, we will build and evaluate our proposed techniques based on a system designed for concurrency research. The system uses the GraalVM technology of Oracle Labs, which allows us to prototype at the level of state-of-the-art systems, while keeping the development effort manageable for a small team.