Young, rapidly-rotating M dwarfs exhibit prominent starspots, which create quasiperiodic signals in their photometric and Doppler spectroscopic measurements. The periodic Doppler signals can mimic radial velocity (RV) changes expected from orbiting exoplanets. Exoplanets can be distinguished from activity-induced false positives by the chromaticity and long-term incoherence of starspot signals, but these qualities are poorly constrained for fully-convective M stars. Coherent photometric starspot signals on M dwarfs may persist for hundreds of rotations, and the wavelength dependence of starspot RV signals may not be consistent between stars due to differences in their magnetic fields and active regions. We obtained precise multi-wavelength RVs of four rapidly-rotating M dwarfs (AD Leo, G 227-22, GJ 1245B, GJ 3959) using the near-infrared (NIR) Habitable-zone Planet Finder, and the optical Keck/HIRES spectrometer. Our RVs are complemented by photometry from Kepler, TESS, and the Las Cumbres Observatory (LCO) network of telescopes. We found that all four stars exhibit large spot-induced Doppler signals at their rotation periods, and investigated the longevity and optical-to-NIR chromaticity for these signals. The phase curves remain coherent much longer than is typical for Sunlike stars. Their chromaticity varies, and one star (GJ 3959) exhibits optical and NIR RV modulation consistent in both phase and amplitude. In general, though, we find that the NIR amplitudes are lower than their optical counterparts. We conclude that starspot modulation for rapidly-rotating M stars frequently remains coherent for hundreds of stellar rotations, and gives rise to Doppler signals that, due to this coherence, may be mistaken for exoplanets.