TrueJIT: Learning and prediction of compilation sequences in a centralized JIT compiler

In recent years, we have witnessed the development of centralized just-in-time (JIT) compilers serving many compiler-less language virtual machine (LVM) clients at once. While centralized JIT compilation reduces CPU and memory overhead due to global code caching and code reuse, it introduces additional latency between LVM clients and the central JIT compilation server. In this paper, we investigate opportunities for hiding this latency. We explore how sequences of JIT compilation requests can be learned and predicted by the centralized JIT compiler. We train a Long Short-Term Memory (LSTM) network with JIT compilation requests and, during deployment, we use this to speculatively compile and deliver native code to the LVM clients in a true just-in-time fashion, i.e. with minimal latency for the LVM clients. We have implemented our novel scheme in a distributed WebAssembly runtime environment, and evaluated it against several benchmark suites including real-world applications from various application domains. We show that learning of JIT compilation sequences is feasible, and prediction is effective in hiding JIT compilation latency. We demonstrate that centralized JIT compilation assisted by code caching, predictive compilation and code delivery reduces JIT compilation latency by, on average, 1-2 orders of magnitude across six benchmark suites and additional complex, real-world applications.