In Paper I of this series, we detected a significant value of the braking index ($n$) for 19 young, high-$\dot{E}$ radio pulsars using $\sim$ 10 years of timing observations from the 64-m Parkes radio telescope. Here we investigate this result in more detail using a Bayesian pulsar timing framework to model timing noise and to perform selection to distinguish between models containing exponential glitch recovery and braking index signatures. We show that consistent values of $n$ are maintained with the addition of substantial archival data, even in the presence of glitches. We provide strong arguments that our measurements are unlikely due to exponential recovery signals from unseen glitches even though glitches play a key role in the evolution of a pulsar's spin frequency. We conclude that, at least over decadal time scales, the value of $n$ can be significantly larger than the canonical 3 and discuss the implications for the evolution of pulsars.