The Jovian Trojans are two swarms of objects located around the L$_4$ and L$_5$ Lagrange points. The population is thought to have been captured by Jupiter during the Solar system's youth. Within the swarms, six collisional families have been identified in previous work, with four in the L$_4$ swarm, and two in the L$_5$. Our aim is to investigate the stability of the two Trojan swarms, with a particular focus on these collisional families. We find that the members of Trojan swarms escape the population at a linear rate, with the primordial L$_4$ (23.35% escape) and L$_5$ (24.89% escape) population sizes likely 1.31 and 1.35 times larger than today. Given that the escape rates were approximately equal between the two Trojan swarms, our results do not explain the observed asymmetry between the two groups, suggesting that the numerical differences are primordial in nature, supporting previous studies. Upon leaving the Trojan population, the escaped objects move onto orbits that resemble those of the Centaur and short-period comet populations. Within the Trojan collisional families, the 1996 RJ and 2001 UV$_{209}$ families are found to be dynamically stable over the lifetime of the Solar system, whilst the Hektor, Arkesilos and Ennomos families exhibit various degrees of instability. The larger Eurybates family shows 18.81\% of simulated members escaping the Trojan population. Unlike the L4 swarm, the escape rate from the Eurybates family is found to increase as a function of time, allowing an age estimation of approximately $1.045\pm 0.364 \times 10^9$ years.