The apparent distribution of large-scale structures in the universe is sensitive to the velocity/potential of the sources as well as the potential along the line-of-sight through the mapping from real space to redshift space (redshift-space distortions, RSD). Since odd multipoles of the halo cross-correlation function vanish when considering standard Doppler RSD, the dipole is a sensitive probe of relativistic and wide-angle effects. We build a catalogue of ten million haloes (Milky-Way size to galaxy-cluster size) from the full-sky light-cone of a new "RayGalGroupSims" N-body simulation which covers a volume of ($2.625~h^{-1}$Gpc)$^3$ with $4096^3$ particles. Using ray-tracing techniques, we find the null geodesics connecting all the sources to the observer. We then self-consistently derive all the relativistic contributions (in the weak-field approximation) to RSD: Doppler, transverse Doppler, gravitational, lensing and integrated Sachs-Wolfe. It allows us, for the first time, to disentangle all contributions to the dipole from linear to non-linear scales. At large scale, we recover the linear predictions dominated by a contribution from the divergence of neighbouring line-of-sights. While the linear theory remains a reasonable approximation of the velocity contribution to the dipole at non-linear scales it fails to reproduce the potential contribution below $30-60~h^{-1}$Mpc (depending on the halo mass). At scales smaller than $\sim 10~h^{-1}$Mpc, the dipole is dominated by the asymmetry caused by the gravitational redshift. The transition between the two regimes is mass dependent as well. We also identify a new non-trivial contribution from the non-linear coupling between potential and velocity terms.