We present high-precision spectro-polarimetric data with high spatial resolution (0.4$''$) of the very quiet Sun at 1.56$\mu$m obtained with the GREGOR telescope to shed some light on this complex magnetism. Half of our observed quiet-Sun region is better explained by magnetic substructure within the resolution element. However, we cannot distinguish whether this substructure comes from gradients of the physical parameters along the line of sight or from horizontal gradients (across the surface). In these pixels, a model with two magnetic components is preferred, and we find two distinct magnetic field populations. The population with the larger filling factor has very weak ($\sim$150 G) horizontal fields similar to those obtained in previous works. We demonstrate that the field vector of this population is not constrained by the observations, given the spatial resolution and polarimetric accuracy of our data. The topology of the other component with the smaller filling factor is constrained by the observations for field strengths above 250 G: we infer hG fields with inclinations and azimuth values compatible with an isotropic distribution. The filling factors are typically below 30\%. We also find that the flux of the two polarities is not balanced. From the other half of the observed quiet-Sun area $\sim$50\% are two-lobed Stokes $V$ profiles, meaning that 23\% of the field of view can be adequately explained with a single constant magnetic field embedded in a non-magnetic atmosphere. The magnetic field vector and filling factor are reliable inferred in only 50\% based on the regular profiles. Therefore, 12\% of the field of view harbour hG fields with filling factors typically below 30\%. At our present spatial resolution, 70\% of the pixels apparently are non-magnetised.