In the Milky Way galaxy, positrons, which are responsible for the diffuse $511\,\mathrm{keV}$ gamma ray emission observed by space-based gamma ray observatories, are thought to annihilate predominantly through charge exchange interactions with neutral hydrogen. These charge exchange interactions can only take place if positrons have energies greater than $6.8\,\mathrm{eV}$, the minimum energy required to liberate the electron bound to the hydrogen atom and then form positronium, a short-lived bound state composed of a positron-electron pair. Here we demonstrate the importance of positron interactions with neutral alkali metals in the warm interstellar medium (ISM). Positrons may undergo charge exchange with these atoms at any energy. In particular, we show that including positron interactions with sodium at solar abundance in the warm ISM can significantly reduce the annihilation timescale of positrons with energies below $6.8\,\mathrm{eV}$ by at least an order of magnitude. We show that including these interactions in our understanding of positron annihilation in the Milky Way rules out the idea that the number of positrons in the Galactic ISM could be maintained in steady state by injection events occurring at a typical periodicity $>\mathrm{Myr}$.