We present a systematic analysis of the mid-infrared (MIR) properties of the youngest radio galaxies, based on low-resolution data provided by the {\it WISE} and {\it IRAS} satellites. We restrict our analysis to sources with available X-ray data that constitute the earliest phase of radio galaxy evolution, i.e. those classified as Gigahertz Peaked Spectrum (GPS) and/or Compact Symmetric Objects (CSOs). In our sample of 29 objects, we find that the host galaxies are predominantly red/yellow ellipticals, with some of them displaying distorted morphology. We find a variety of MIR colors, and observe that the sources in which the MIR emission is dominated by the ISM component uniformly populate the region occupied by galaxies with a wide range of pronounced ($\geq 0.5 M_{\odot}$\,yr$^{-1}$) star formation activity. We compare the MIR color distribution in our sample to that in the general population of local AGN, in the population of evolved FR\,II radio galaxies, and also in the population of radio galaxies with recurrent jet activity. We conclude that the triggering of radio jets in AGN does not differentiate between elliptical hosts with substantially different fractions of young stars; instead there is a relationship between the jet duty cycle and the ongoing star formation. The distribution of the sub-sample of our sources with $z<0.4$ on the low-resolution MIR vs. absorption-corrected X-ray luminosity plane is consistent with the distribution of a sample of local AGN. Finally, we comment on the star formation rates of the two $\gamma$-ray detected sources in our sample, 1146+596 \& 1718--649.