The detection of the pulsed $\sim 1 $~TeV gamma-ray emission from the Crab pulsar reported by MAGIC and VERITAS collaborations demands a substantial revision of existing models of particle acceleration in the pulsar magnetosphere. In this regard model independent restrictions on the possible production site of the VHE photons become an important issue. In this paper, we consider limitations imposed by the process of conversion of VHE gamma rays into $e^{\pm}$ pairs in the magnetic field of the pulsar magnetosphere. Photons with energies exceeding 1~TeV are effectively absorbed even at large distances from the surface of the neutron star. Our calculations of magnetic absorption in the force-free magnetosphere show that the twisting of the magnetic field due to the pulsar rotation makes the magnetosphere more transparent compared to the dipole magnetosphere. The gamma-ray absorption appears stronger for photons emitted in the direction of rotation than in the opposite direction. There is a small angular cone inside which the magnetosphere is relatively transparent and photons with energy $1.5$~TeV can escape from distances beyond $0.1$~light cylinder radius ($R_{\rm{lc}}$). The emission surface from where photons can be emitted in the observer's direction further restricts the sites of VHE gamma-ray production. For the observation angle $57^{\circ}$ relative to the Crab pulsar axis of rotation and the orthogonal rotation, the emission surface in the open field line region is located as close as $0.4\,R_{\rm{lc}}$ from the stellar surface for a dipole magnetic field, and $0.1\,R_{\rm{lc}}$ for a force-free magnetic field.