We investigate the stellar kinematics and stellar populations of 58 radio-loud galaxies of intermediate luminosities (L$_{3 GHz}$ $>$ 10$^{23}$ W Hz$^{-1}$ ) at 0.6 < z < 1. This sample is constructed by cross-matching galaxies from the deep VLT/VIMOS LEGA-C spectroscopic survey with the VLA 3 GHz dataset. The LEGA-C continuum spectra reveal for the first time stellar velocity dispersions and age indicators of z $\sim$ 1 radio galaxies. We find that $z\sim 1$ radio-loud AGN occur exclusively in predominantly old galaxies with high velocity dispersions: $\sigma_*>$ 175 km s$^{-1}$, corresponding to black hole masses in excess of $10^8$ M$_{\odot}$. Furthermore, we confirm that at a fixed stellar mass the fraction of radio-loud AGN at z $\sim$ 1 is 5 - 10 times higher than in the local universe, suggesting that quiescent, massive galaxies at z $\sim$ 1 switch on as radio AGN on average once every Gyr. Our results strengthen the existing evidence for a link between high black-hole masses, radio loudness and quiescence at z $\sim$ 1.