The evolution of neutron star (NS) magnetic field (B-field) has long been an important topic, which is still not yet settled down. Here, we analyze the NS B-fields inferred by the cyclotron resonance scattering features (CRSFs) for the high mass X-ray binaries (HMXBs) and by the magnetic dipole model for the spin-down pulsars. We find that the B-fields of both the 32 NSHMXBs and 28 young pulsars with the supernova remnants follow the log-normal distributions, with the average values of 3.4 * 10^12 G and 4.1 * 10^12 G respectively, which are further verified to come from the same continuous distribution by the statistical tests. These results declaim that the two methods of measuring NS B-fields are reliable for the above two groups of samples. In addition, since the NS-HMXBs have experienced the spin-down phase as the normal pulsars without accretion and then the spin-up phase by accretion, their ages should be about million years (Myrs). Our statistical facts imply that the B-fields of NS-HMXBs have little decayed in their non-accretion spin-down phases of ~ Myrs, as well as in their accretion phases of ~ 0.1Myrs.