A large fraction of Gamma Ray Bursts (GRBs) displays an X-ray plateau phase within <10^{5} s from the prompt emission, proposed to be powered by the spin-down energy of a rapidly spinning newly born magnetar. In this work we use the properties of the Galactic neutron star population to constrain the GRB-magnetar scenario. We re-analyze the X-ray plateaus of all Swift GRBs with known redshift, between January 2005 and August 2014. From the derived initial magnetic field distribution for the possible magnetars left behind by the GRBs, we study the evolution and properties of a simulated GRB-magnetar population using numerical simulations of magnetic field evolution, coupled with Monte Carlo simulations of Pulsar Population Synthesis in our Galaxy. We find that if the GRB X-ray plateaus are powered by the rotational energy of a newly formed magnetar, the current observational properties of the Galactic magnetar population are not compatible with being formed within the GRB scenario (regardless of the GRB type or rate at z=0). Direct consequences would be that we should allow the existence of magnetars and "super-magnetars" having different progenitors, and that Type Ib/c SNe related to Long GRBs form systematically neutron stars with higher initial magnetic fields. We put an upper limit of <16 "super-magnetars" formed by a GRB in our Galaxy in the past Myr (at 99% c.l.). This limit is somewhat smaller than what roughly expected from Long GRB rates, although the very large uncertainties do not allow us to draw strong conclusion in this respect.