Using isochronous mass spectrometry at the experimental storage ring CSRe in Lanzhou, the masses of $^{82}$Zr and $^{84}$Nb were measured for the first time with an uncertainty of $\sim 10$ keV, and the masses of $^{79}$Y, $^{81}$Zr, and $^{83}$Nb were re-determined with a higher precision. %The latter differ significantly from their literature values. The latter are significantly less bound than their literature values. Our new and accurate masses remove the irregularities of the mass surface in this region of the nuclear chart. Our results do not support the predicted island of pronounced low $\alpha$ separation energies for neutron-deficient Mo and Tc isotopes, making the formation of Zr-Nb cycle in the $rp$-process unlikely. The new proton separation energy of $^{83}$Nb was determined to be 490(400)~keV smaller than that in the Atomic Mass Evaluation 2012. This partly removes the overproduction of the $p$-nucleus $^{84}$Sr relative to the neutron-deficient molybdenum isotopes in the previous $\nu p$-process simulations.