Level Graphs: Generating Benchmarks for Concurrency Optimizations in Compilers

Benchmarks are needed in order to test compiler and languagebased approaches to optimize concurrency. These have to be varied, yield reproducible results and allow comparison between different approaches. In this paper, we propose a framework for generating synthetic benchmarks that aims at attaining these goals. Based on generating code from random graphs, our framework operates at a high level of abstraction. We test our benchmarking framework with a usecase, where we compare three state-of-the-art systems to optimize I/O concurrency in microservice-based software architectures. We show how using our benchmarks we can reliably compare between approaches, and even between the same approach using different coding styles.