CaMELot: Catching and Mitigating Event-Loop Concurrency Issues

大多数现代计算机应用程序以这样或那样的方式依赖于由因特网上的服务器应用程序执行的计算。越来越多的这些服务器应用程序现在被构建为所谓的微服务，它允许开发人员逐步更新或修复较大应用程序的不相关部分中的问题，因此变得流行起来。这些微服务中的许多在设计上都避免了某些类型的并发问题。不幸的是，他们仍然受到其他类型的并发问题的困扰，例如当多个在线客户试图同时预订相同的座位时。对于软件工程师来说，很难测试所有可能的并发交互。在实践中，这意味着在测试期间只可靠地检测到简单的并发问题。然而，复杂的问题很容易被忽略并进入服务器应用程序，然后错误地处理客户端请求。当Facebook股票第一次交易时，这种并发问题的一个例子出现在纳斯达克，导致数百万美元的损失。我们的目标是开发在运行时自动检测并发问题的技术，能够绕过它们，并使开发人员能够使用检测技术收集的详细信息来修复它们。研究人员已经证明，人们可以检测并避免问题，例如，通过改变处理客户请求的顺序。然而，在实践中，当前的技术显著降低了服务器应用的速度，这使得这些技术的使用成本太高。我们的目标是动态平衡对准确信息的需求，并将速度减慢降至最低。我们猜测，我们可以获得最实际的好处，而只需很少跟踪程序代码如何执行的准确细节。除了自动防止并发问题导致问题外，我们还将使用获得的信息向开发人员提供反馈，以便他们可以修复软件中的潜在问题。因此，总体而言，此研究项目的目标是使服务器应用程序，特别是微服务，对软件错误更健壮和更具弹性，这些错误很难测试，因此通常保持未被发现，直到它们给客户或公司造成重大问题。我们的工作将导致开发自适应技术来检测并发问题，并自动权衡精确度和运行时开销，以便在实践中使用。此外，检测技术将被用来为软件开发人员提供可操作的输入，以便修复并发问题，从而在未来可靠地防止并发问题。为了评估这项工作，我们将收集各种不同类型的并发问题并将其公开提供。此集合将基于工业系统中的问题，并从高度复杂的错误的理论场景中派生出来。我们包括这些理论场景，因为这些复杂的错误很难诊断和测试，它们在实践中可能仍然没有诊断和文档记录，但有可能导致重大中断。最后，我们将基于为并发研究设计的系统来构建和评估我们提出的技术。该系统使用Oracle Labs的GraalVM技术，它允许我们在最先进的系统级别上进行原型制作，同时使开发工作对于一个小团队来说是可管理的。

Most modern computer applications depend in some way or another on computations that are performed by server applications on the internet. More and more of these server applications are now built as so-called microservices, which allow developers to gradually update or fix issues in unrelated parts of a larger application, and therefore, have become popular. Many of these microservices avoid certain types of concurrency issues by design. Unfortunately, they still suffer from other kinds of concurrency issues, for example when multiple online customers try to reserve the same seats at the same time.For software engineers, it is hard to test for all possible concurrent interactions. In practice, this means that only simple concurrency issues are reliably detected during testing. Complex issues can however easily slip through and make it into server applications and then handle client requests incorrectly. One example of such a concurrency issue appeared at Nasdaq when the Facebook stock was traded for the first time, resulting in the loss of millions of dollars.Our goal is to develop techniques that detect concurrency issues automatically at run time, to be able to circumvent them, and enable developers to fix them, using detailed information gathered by the detection techniques. Researchers have shown that one can detect and avoid issues, for instance by changing the order in which client requests are processed. In practice however, current techniques slow server applications down significantly, which make these techniques too costly to be used. Our aim is to dynamically balance the need for accurate information and minimize slow down. We conjecture that we can get most practical benefits while only rarely tracking precise details of how program code executes. In addition to automatically preventing concurrency issues to cause problems, we will also use the obtained information to provide feedback to developers so that they can fix the underlying issue in their software.Thus, overall the goal of this research project is to make server applications, and specifically microservices, more robust and resilient to software bugs that are hard to test for and therefore typically remain undiscovered until they cause major issues for customers or companies.Our work will result in the development of adaptive techniques that detect concurrency issues, and automatically tradeoff accuracy and run-time overhead, to be usable in practice. Furthermore, the detection techniques will be used to provide actionable input to the software developers, so that the concurrency issue can be fixed and therefore be prevented reliably in the future.To evaluate this work, we will collect various different types of concurrency issues and make them openly available. This collection will be based on issues from industrial systems and derived from theoretical scenarios for highly complex bugs. We include these theoretical scenarios, since such complex bugs are hard to diagnose and test for, they likely remain undiagnosed and undocumented in practice, but have the potential of causing major disruptions.Finally, we will build and evaluate our proposed techniques based on a system designed for concurrency research. The system uses the GraalVM technology of Oracle Labs, which allows us to prototype at the level of state-of-the-art systems, while keeping the development effort manageable for a small team.