In this letter we calculate the energy loss of highly magnetized neutron star due to friction with quantum vacuum, namely Quantum Vacuum Friction (QVF). Taking into account one-loop corrections in the effective Heisenberg-Euler Lagrangian of the light-light interaction, we derive an analytic expression for QVF allowing us to consider magnetic field at the surface of the star as high as $10^{11} $T. In the case of magnetars with high magnetic field above the QED critical field, we show that the energy loss by QVF dominates the energy loss process. This has important consequences, in particular on the inferred value of the magnetic field. This also indicates the need for independent measurements of magnetic field, energy loss rate, and of the braking index to fully characterize magnetars.