The kinetic Sunyaev Zel'dovich (kSZ) effect, cosmic microwave background (CMB) temperature anisotropies induced by scattering of CMB photons from free electrons, forms the dominant blackbody component of the CMB on small angular scales. Future CMB experiments on the resolution/sensitivity frontier will measure the kSZ effect with high significance. Through the cross-correlation with tracers of structure, it will be possible to reconstruct the remote CMB dipole field (e.g. the CMB dipole observed at different locations in the Universe) using the technique of kSZ tomography. In this paper, we derive a quadratic estimator for the remote dipole field using the Cosmic Infrared Background (CIB) as a tracer of structure. We forecast the ability of current and next-generation measurements of the CMB and CIB to perform kSZ tomography. The total signal-to-noise of the reconstruction is order unity for current datasets, and can improve by a factor of up to $10^3$ for future datasets, based on statistical error only. The CIB-based reconstruction is highly correlated with a galaxy survey-based reconstruction of the remote dipole field, which can be exploited to improve constraints on cosmological parameters and models for the CIB and distribution of baryons.