We study the displacements between the centres of galaxies and their supermassive black holes (BHs) in the cosmological hydrodynamical simulation Horizon-AGN, and in a variety of observations from the literature. The BHs in Horizon-AGN feel a sub-grid dynamical friction force, sourced by the surrounding gas, which prevents recoiling BHs being ejected from the galaxy. We find that i) the fraction of spatially offset BHs increases with cosmic time, ii) BHs live on prograde orbits in the plane of the galaxy with an orbital radius that decays with time but stalls near $z=0$, and iii) the magnitudes of offsets from the galaxy centres are substantially larger in the simulation than in observations. We attribute the stalling of the infall and excessive offset magnitudes to the fact that dynamical friction from stars and dark matter is not modelled in the simulation, and hence provide a way to improve the black hole dynamics of future simulations.