The limiting magnitude is a key issue for optical interferometry. Pairwise fringe trackers based on the integrated optics concepts used for example in GRAVITY seem limited to about K=10.5 with the 8m Unit Telescopes of the VLTI, and there is a general "common sense" statement that the efficiency of fringe tracking, and hence the sensitivity of optical interferometry, must decrease as the number of apertures increases, at least in the near infrared where we are still limited by detector readout noise. Here we present a Hierarchical Fringe Tracking (HFT) concept with sensitivity at least equal to this of a two apertures fringe trackers. HFT is based of the combination of the apertures in pairs, then in pairs of pairs then in pairs of groups. The key HFT module is a device that behaves like a spatial filter for two telescopes (2TSF) and transmits all or most of the flux of a cophased pair in a single mode beam. We give an example of such an achromatic 2TSF, based on very broadband dispersed fringes analyzed by grids, and show that it allows piston measures from very broadband fringes with only 3 to 5 pixels per fringe tracker. We show the results of numerical simulation indicating that our device is a good achromatic spatial filter and allowing a first evaluation of its coupling efficiency, which is similar to this of a single mode fiber on a single aperture. Our very preliminary results indicate that HFT has a good chance to be a serious candidate for the most sensitive fringe tracking with the VLTI and also interferometers with much larger number of apertures. On the VLTI the first rough estimate of the magnitude gain with regard to the GRAVITY internal FT is between 2.5 and 3.5 magnitudes in K, with a decisive impact on the VLTI science program for AGNs, Young stars and planet forming disks.