Theoretical arguments in favor of energy dependent photon time delays from a modification of special relativity (SR) have met with recent gamma ray observations that put severe constraints on the scale of such deviations. We review the case of the generality of this theoretical prediction in the case of a deformation of SR and find that, at least in the simple model based on the analysis of photon worldlines which is commonly considered, there are many scenarios compatible with a relativity principle which do not contain a photon time delay. This will be the situation for any modified dispersion relation which reduces to $E=|\vec{p}|$ for photons, independently of the quantum structure of spacetime. This fact opens up the possibility of a phenomenologically consistent relativistic generalization of SR with a new mass scale many orders of magnitude below the Planck mass.