Sunset solar axions traversing the intense magnetic field of the CERN Axion Solar Telescope (CAST) experiment may be detected in a Time Projection Chamber (TPC) detector, as X-rays signals. These signals could be masked, however, by the inhomogeneous background of materials in the experimental site. A detailed analysis, based on the detector characteristics, the background radiation at the CAST site, simulations and experimental results, has allowed us to design a shielding which reduces the background level by a factor of ~4 compared to the detector without shielding, depending on its position, in the energy range between 1 and 10 keV. Moreover, this shielding has improved the homogeneity of background measured by the TPC.