The surface temperatures as well as X-ray luminosities of Anomalous X-ray Pulsars and Soft Gamma Repeaters are several times higher than that of the ordinary isolated neutron stars at similar ages. We present a simple approach to explain this observation by the effect of the neutrino magnetic moment on neutron star cooling. This requires a neutrino magnetic moment about $(3-5)\times10^{-11}\mu_B$. The simple approach presented in this paper might be used in determining the exact value of the neutrino magnetic moment and its contribution to the neutrino-electron scattering when the models to explain Anomalous X-ray Pulsars and Soft Gamma Repeaters are mature enough.