Radio telescopes are used to accurately measure the time of arrival (ToA) of radio pulses in pulsar timing experiments that target mostly millisecond pulsars (MSPs) due to their high rotational stability. This allows for detailed study of MSPs and forms the basis of experiments to detect gravitational waves. Apart from intrinsic and propagation effects, such as pulse-to-pulse jitter and dispersion variations in the interstellar medium, timing precision is limited in part by the following: polarization purity of the telescope's orthogonally polarized receptors, the signal-to-noise ratio (S/N) of the pulsar profile, and the polarization fidelity of the system. Using simulations, we present how fundamental limitations in recovering the true polarization reduce the precision of ToA measurements. Any real system will respond differently to each source observed depending on the unique pulsar polarization profile. Using the profiles of known MSPs we quantify the limits of observing system specifications that yield satisfactory ToA measurements, and we place a practical design limit beyond which improvement of the system results in diminishing returns. Our aim is to justify limits for the front-end polarization characteristics of next generation radio telescopes, leading to the Square Kilometre Array (SKA).