We combine a deep 0.5~deg$^2$, 1.4~GHz deep radio survey in the Lockman Hole with infrared and optical data in the same field, including the SERVS and UKIDSS near-infrared surveys, to make the largest study to date of the host galaxies of radio sources with typical radio flux densities $\sim 50 \;\mu$Jy. 87% (1274/1467) of radio sources have identifications in SERVS to $AB\approx 23.1$ at 3.6 or 4.5$\mu$m, and 9% are blended with bright objects (mostly stars), leaving only 4% (59 objects) which are too faint to confidently identify in the near-infrared. We are able to estimate photometric redshifts for 68% of the radio sources. We use mid-infrared diagnostics to show that the source population consists of a mixture of star forming galaxies, rapidly accreting (cold mode) AGN and low accretion rate, hot mode AGN, with neither AGN nor starforming galaxies clearly dominating. We see the breakdown in the $K-z$ relation in faint radio source samples, and show that it is due to radio source populations becoming dominated by sources with radio luminosities $\sim 10^{23}\;{\rm WHz^{-1}}$. At these luminosities, both the star forming galaxies and the cold mode AGN have hosts with stellar luminosities about a factor of two lower than those of hot mode AGN, which continue to reside in only the most massive hosts. We show that out to at least $z\sim 2$, galaxies with stellar masses $>10^{11.5}\, M_{\odot}$ have a radio-loud fraction up to $\sim 30$%. This is consistent with there being a sufficient number of radio sources that radio-mode feedback could play a role in galaxy evolution.