The NIRC2 vortex coronagraph is an instrument on Keck II designed to directly image exoplanets and circumstellar disks at mid-infrared bands $L^\prime$ (3.4-4.1 $\mu$m) and $M_s$ (4.55-4.8 $\mu$m). We analyze imaging data and corresponding adaptive optics telemetry, observing conditions, and other metadata over a three year time period to characterize the performance of the instrument and predict the detection limits of future observations. We systematically process images from 359 observations of 304 unique stars to subtract residual starlight (i.e., the coronagraphic point spread function) of the target star using two methods: angular differential imaging (ADI) and reference star differential imaging (RDI). We find that for the typical parallactic angle (PA) rotation of our dataset ($\sim$10$^{\circ}$), RDI provides gains over ADI for angular separations smaller than 0.25$^{\prime\prime}$. Furthermore, we find a power-law relation between the angular separation from the host star and the minimum PA rotation required for ADI to outperform RDI, with a power-law index of -1.18$\pm$0.08. Finally, we use random forest models to estimate ADI and RDI post-processed detection limits a priori. These models, which we provide publicly on a website, explain 70%-80% of the variance in ADI detection limits and 30%-50% of the variance in RDI detection limits. Averaged over a range of angular separations, our models predict both ADI and RDI contrast to within a factor of 2. These results illuminate important factors in high-contrast imaging observations with the NIRC2 vortex coronagraph, help improve observing strategies, and inform future upgrades to the hardware.