Dark photon dark matter will resonantly convert into visible photons when the dark photon mass is equal to the plasma frequency of the ambient medium. In cosmological contexts, this transition leads to an extremely efficient, albeit short-lived, heating of the surrounding gas. Existing work in this field has been predominantly focused on understanding the implications of these resonant transitions in the limit that the plasma frequency of the Universe can be treated as being perfectly homogeneous, \ie neglecting inhomogeneities in the electron number density. In this work we focus on the implications of heating from dark photon dark matter in the presence of inhomogeneous structure (which is particularly relevant for dark photons with masses in the range $10^{-15} \, {\rm eV} \, \lesssim m_{A^\prime} \lesssim 10^{-12}$ eV), emphasizing both the importance of inhomogeneous energy injection, as well as the sensitivity of cosmological observations to the inhomogeneities themselves. More specifically, we derive modified constraints on dark photon dark matter from the Ly-$\alpha$ forest, and show that the presence of inhomogeneities allows one to extend constraints to masses outside of the range that would be obtainable in the homogeneous limit, while only slightly relaxing their strength. We then project sensitivity for near-future cosmological surveys that are hoping to measure the 21cm transition in neutral hydrogen prior to reionization, and demonstrate that these experiments will be extremely useful in improving sensitivity to masses near $\sim 10^{-14}$ eV, potentially by several orders of magnitude. Finally, we discuss implications for reionization, early star formation, and late-time $y$-type spectral distortions, and show that probes which are inherently sensitive to the inhomogeneous state of the Universe could resolve signatures unique to the light dark photon...