We present high-precision spectro-polarimetric data with high spatial resolution (0.4$''$) of the very quiet Sun at 1.56$\mu$m obtained with the GREGOR telescope to shed some light on this complex magnetism. Half of our observed quiet-Sun region is better explained by magnetic substructure within the resolution element. However, we cannot distinguish whether this substructure comes from gradients of the physical parameters along the line of sight or from horizontal gradients (across the surface). In these pixels, a model with two magnetic components is preferred, and we find two distinct magnetic field populations. The population with the larger filling factor has very weak ($\sim$150 G) horizontal fields similar to those obtained in previous works. We demonstrate that the field vector of this population is not constrained by the observations, given the spatial resolution and polarimetric accuracy of our data. The topology of the other component with the smaller filling factor is constrained by the observations for field strengths above 250 G: we infer hG fields with inclinations and azimuth values compatible with an isotropic distribution. The filling factors are typically below 30\%. We also find that the flux of the two polarities is not balanced. From the other half of the observed quiet-Sun area $\sim$50\% are two-lobed Stokes $V$ profiles, meaning that 23\% of the field of view can be adequately explained with a single constant magnetic field embedded in a non-magnetic atmosphere. The magnetic field vector and filling factor are reliable inferred in only 50\% based on the regular profiles. Therefore, 12\% of the field of view harbour hG fields with filling factors typically below 30\%. At our present spatial resolution, 70\% of the pixels apparently are non-magnetised.

Context. A significant part of the penumbral magnetic field returns below the surface in the very deep photosphere. For lines in the visible, a large portion of this return field can only be detected indirectly by studying its imprints on strongly asymmetric and three-lobed Stokes V profiles. Infrared lines probe a narrow layer in the very deep photosphere, providing the possibility of directly measuring the orientation of magnetic fields close to the solar surface. Aims. We study the topology of the penumbral magnetic field in the lower photosphere, focusing on regions where it returns below the surface. Methods. We analyzed 71 spectropolarimetric datasets from Hinode and from the GREGOR infrared spectrograph. We inferred the quality and polarimetric accuracy of the infrared data after applying several reduction steps. Techniques of spectral inversion and forward synthesis were used to test the detection algorithm. We compared the morphology and the fractional penumbral area covered by reversed-polarity and three-lobed Stokes V profiles for sunspots at disk center. We determined the amount of reversed-polarity and three-lobed Stokes V profiles in visible and infrared data of sunspots at various heliocentric angles. From the results, we computed center-to-limb variation curves, which were interpreted in the context of existing penumbral models. Results. Observations in visible and near-infrared spectral lines yield a significant difference in the penumbral area covered by magnetic fields of opposite polarity. In the infrared, the number of reversed-polarity Stokes V profiles is smaller by a factor of two than in the visible. For three-lobed Stokes V profiles the numbers differ by up to an order of magnitude.