We present the first broadband X-ray observations of four Giga-Hertz Peaked Spectrum (GPS) radio galaxies at redshift <~1 performed by Chandra and XMM-Newton. These observations more than double the number of members of this class with measured spectra in hard (E > 2 keV) X-rays. All sources were detected. Their radio-to-X-ray spectral energy distributions are similar, except for PKS0941-080, which is X-ray under-luminous by about two orders of magnitude. The comparison between the full sample of GPS galaxies with measurements in hard X-rays and a control sample of radio galaxies rules out intrinsic X-ray weakness as causing a lower detection rate of GPS sources in X-ray surveys. Four out of seven GPS galaxies exhibit high X-ray column densities, whereas for the remaining three this measurement is hampered by the poor spectral statistics. Bearing in mind the low number statistics in both the GPS and the control sample, the average column density measured in GPS galaxies is larger than in FRI or Broad Line Region FR II radio galaxies, but consistent with that measured in High-Excitation FRII galaxies. This leads to a location the absorbing gas in an obscuring ``torus'', which prevents us from observing the nuclear region along lines-of-sight perpendicular to the radio axis. This interpretation is supported by the discovery of rapid (timescale ~1000 s) X-ray variability in the GPS galaxy COINSJ0029+3456, and by an almost order-of-magnitude difference between the HI column density measured in radio and X-rays in PKS0500+019.