We demonstrate that a natural consequence of an asymmetric kick imparted to neutron stars at birth is that the majority of double neutron star binaries should possess highly eccentric orbits. This leads to greatly accelerated orbital decay, due to the enormous increase in the emission of gravitational radiation at periastron as originally demonstrated by Peters (1964). A uniform distribution of kick velocities constrained to the orbital plane would result in ~24% of surviving binaries coalescing at least 10,000 times faster than an unperturbed circular system. Even if the planar kick constraint is lifted, ~6% of bound systems still coalesce this rapidly. In a non-negligible fraction of cases it may even be possible that the system could coalesce within 10 years of the final supernova, resulting in what we might term a "double supernova''. For systems resembling the progenitor of PSR J0737-3039A, this number is as high as \~9% (in the planar kick model). Whether the kick velocity distribution extends to the range required to achieve this is still unclear. We do know that the observed population of binary pulsars has a deficit of highly eccentric systems at small orbital periods. In contrast, the long-period systems favour large eccentricities, as expected. We argue that this is because the short-period highly eccentric systems have already coalesced and are thus selected against by pulsar surveys. This effect needs to be taken into account when using the scale-factor method to estimate the coalescence rate of double neutron star binaries. We therefore assert that the coalesence rate of such binaries is underestimated by a factor of several.