Language implementation frameworks such as Truffle+Graal and RPython make the promise of state-of-the-art performance by implementing “just” the interpreter, and leaving the rest to the frameworks, which add a just-in-time compiler, garbage collection, and various other bits “for free”. One important assumption for these frameworks is that real systems do not spend a lot of time interpreting user code, but reach highly-optimized compiled code quickly.

Unfortunately, for large codebases with millions of lines of code, this assumption does not hold as well as for common benchmarks. A significant amount of time is spent interpreting code. This is only exacerbated by modern development approaches, which lead to, what one would assume to be long running server applications, being updated every 30 minutes. In practice, this means for large and actively developed codebases, interpreter performance is key.

This brings us to the question of how Truffle-based interpreters such as Graal.js, TruffleRuby, GraalPython, and TruffleSOM compare to commonly used interpreter implementations for the same language. We will present our results comparing these interpreters with and without just-in-time compilation on the Are We Fast Yet benchmarks, which were designed for cross-language comparison.

We will further analyze where these interpreters spend their time, and experiment with an approach to approximate “best case” performance assuming an interpreter could perform optimizations on the method level without requiring just-in-time compilation.

Based on our observations, we will discuss a number of possible steps forward based on the idea of supernodes, i.e., node combination, object inlining, and generating interpreters using Graal’s partial evaluator. All these techniques attempt to mitigate the performance cost of the “everything is a node” implementation style of Truffe interpreters, which leads to costly run-time program representation and a high degree of redundancy in correctness check during interpretation.