Single-mode fiber (SMF) spectrographs fed with adaptive optics (AO) offer a unique path for achieving extremely precise radial velocity (EPRV) measurements. We present a radial velocity (RV) error budget based on end-to-end numerical simulations of an instrument named iLocater that is being developed for the Large Binocular Telescope(LBT). Representing the first AO-fed, SMF spectrograph, iLocater's design is used to quantify and assess the relative advantages and drawbacks of precise Doppler time series measurements made at the diffraction limit. This framework can be applied for trade-study work to investigate the impact of instrument design decisions on systematic uncertainties encountered in the regime of sub-meter-per-second precision. We find that working at the diffraction-limit through the use of AO and SMF's allows for high spectral resolution and improved instrument stability at the expense of limiting magnitude and longer integration times. Large telescopes equipped with AO alleviates the primary challenges of SMF spectrographs.