Using the simultaneous Infra-Red (IR) and X-ray light curves obtained by Kalamkar et al. (2016), we perform a Fourier analysis of the IR/X-ray timing correlations of the black hole X-ray binary (BHB) GX 339-4. The resulting IR vs X-ray Fourier coherence and lag spectra are similar to those obtained in previous studies of GX 339-4 using optical light curves. In particular, above 1 Hz, the lag spectrum features an approximately constant IR lag of about 100 ms. We model simultaneously the radio to IR Spectral Energy Distribution (SED), the IR Power Spectral Density (PSD), and the coherence and lag spectra using the jet internal shock model ISHEM assuming that the fluctuations of the jet Lorentz factor are driven by the accretion flow. It turns out that most of the spectral and timing features, including the 100 ms lag, are remarkably well reproduced by this model. The 100 ms time-scale is then associated with the travel time from the accretion flow to the IR emitting zone. Our exploration of the parameter space favours a jet which is at most mildly relativistic ($\bar{\Gamma}< 3$), and a linear and positive relation between the jet Lorentz factor and X-ray light curve i.e. $\Gamma(t)-1\propto L_{X}(t)$. The presence of a strong Low Frequency Quasi Periodic Oscillation (LFQPO) in the IR light curve could be caused by jet precession driven by Lense-Thirring precession of the jet-emitting accretion flow. Our simulations confirm that this mechanism can produce an IR LFQPO similar to that observed in GX 339-4.