Aims. We investigate the influence of the Yarkovsky force on the long-term orbital evolution of Jupiter Trojan asteroids. Methods. Clones of the observed population with different sizes and different thermal properties were numerically integrated for 1 Gyr with and without the Yarkovsky effect. The escape rate of these objects from the Trojan region as well as changes in the libration amplitude, eccentricity, and inclination were used as a metric of the strength of the Yarkovsky effect on the Trojan orbits. Results. Objects with radii $R\leq$1 km are significantly influenced by the Yarkovsky force. The effect causes a depletion of these objects over timescales of a few hundred million years. As a consequence, we expect the size-frequency distribution of small Trojans to show a shallower slope than that of the currently observable population ($R$ $\gtrsim$ 1 km), with a turning point between $R$ = 100 m and $R$ = 1 km. The effect of the Yarkovsky acceleration on the orbits of Trojans depends on the sense of rotation in a complex way. The libration amplitude of prograde rotators decreases with time while the eccentricity increases. Retrograde rotators experience the opposite effect, which results in retrograde rotators being ejected faster from the 1:1 resonance region. Furthermore, for objects affected by the Yarkovsky force, we find indications that the effect tends to smooth out the differences in the orbital distribution between the two clouds.