Paper Title
ENHANCING LLVM’S JUST-IN-TIME COMPILATION: A GENERAL-PURPOSE, TARGET-INDEPENDENT IMPLEMENTATION OF ON-STACK REPLACEMENT (OSR)

Abstract
This paper presents a novel, general-purpose, targetindependent implementation of On-Stack Replacement (OSR) for LLVM’s Just-In-Time (JIT) compilation infrastructure. OSR is a dynamic optimization technique that allows a running program to transition seamlessly between different versions of a function, typically in virtual machines that execute dynamic languages. While LLVM’s MCJIT compiler supports advanced features like native code loading and linking, it lacks builtin OSR capabilities, a gap this paper aims to fill. I propose OSRKit, a prototype library designed to implement OSR entirely at the intermediate representation (IR) level, avoiding complex machine-code manipulation. This approach enhances platform independence and enables dynamic optimizations such as deoptimization and runtime-based profiling. My solution supports transitions between optimized and deoptimized function variants at arbitrary points in the execution, providing flexibility and improved performance. I demonstrate the efficacy of OSRKit through a case study on MATLAB’s feval construct, showing how OSR-enabled specialization leads to significant performance gains. Furthermore, my approach incurs minimal intrusiveness, allowing for efficient native code generation via LLVM’s optimization pipeline. An experimental evaluation validates the effectiveness of OSRKit in terms of performance and its low impact on code quality. This work opens up new possibilities for dynamic runtime optimizations and highlights the potential of OSR for variome JIT-compiled languages, including Python, Ruby, and Julia.