The interpretation of upcoming weak gravitational lensing surveys depends critically on our understanding of the matter power spectrum on scales $k < 10 h/\mathrm{Mpc}$, where baryonic processes are important. We study the impact of galaxy formation processes on the matter power spectrum using a halo model that treats the stars and gas separately from the dark matter distribution. We use empirical constraints from X-ray observations (hot gas) and halo occupation distribution modelling (stars) for the baryons. Since X-ray observations cannot generally measure the hot gas content outside $r_\mathrm{500c}$, we vary the gas density profiles beyond this radius. Compared with dark matter only models, we find a total power suppression of $1 \%$ ($5 \%$) on scales $0.2-1 h/\mathrm{Mpc}$ ($0.5-2h/\mathrm{Mpc}$), where lower baryon fractions result in stronger suppression. We show that groups of galaxies ($10^{13} < m_\mathrm{500c} / (\mathrm{M_\odot}/h) < 10^{14}$) dominate the total power at all scales $k \lesssim 10 h/\mathrm{Mpc}$. We find that a halo mass bias of $30 \%$ (similar to what is expected from the hydrostatic equilibrium assumption) results in an underestimation of the power suppression of up to $4 \%$ at $k = 1 h/\mathrm{Mpc}$, illustrating the importance of measuring accurate halo masses. Contrary to work based on hydrodynamical simulations, our conclusion that baryonic effects can no longer be neglected is not subject to uncertainties associated with our poor understanding of feedback processes. Observationally, probing the outskirts of groups and clusters will provide the tightest constraints on the power suppression for $k \lesssim 1 h/\mathrm{Mpc}$.