Metric perturbations affect the phase of gravitational waves as they propagate through the inhomogeneous universe. This effect causes Stochastic Gravitational Wave Backgrounds (SGWBs) to lose any phase coherence that may have been present at emission or horizon entry. We show that, for a standard cosmological model, this implies complete loss of coherence above frequencies $f \sim 10^{-12}$ Hz. The result is that any attempts to map SGWBs using phase-coherent methods have no foreseeable applications. Incoherent methods that solve directly for the intensity of the SGWBs are the only methods that can reconstruct the angular dependence of any SGWB.