Recent work has suggested that mid-IR wavelengths are optimal for estimating the mass-to-light ratios of stellar populations and hence the stellar masses of galaxies. We compare stellar masses deduced from spectral energy distribution (SED) models, fitted to multi-wavelength optical-NIR photometry, to luminosities derived from {\it WISE} photometry in the $W1$ and $W2$ bands at 3.6 and 4.5$\mu$m for non-star forming galaxies. The SED derived masses for a carefully selected sample of low redshift ($z \le 0.15$) passive galaxies agree with the prediction from stellar population synthesis models that $M_*/L_{W1} \simeq 0.6$ for all such galaxies, independent of other stellar population parameters. The small scatter between masses predicted from the optical SED and from the {\it WISE} measurements implies that random errors (as opposed to systematic ones such as the use of different IMFs) are smaller than previous, deliberately conservative, estimates for the SED fits. This test is subtly different from simultaneously fitting at a wide range of optical and mid-IR wavelengths, which may just generate a compromise fit: we are directly checking that the best fit model to the optical data generates an SED whose $M_*/L_{W1}$ is also consistent with separate mid-IR data. We confirm that for passive low redshift galaxies a fixed $M_*/L_{W1} = 0.65$ can generate masses at least as accurate as those obtained from more complex methods. Going beyond the mean value, in agreement with expectations from the models, we see a modest change in $M_*/L_{W1}$ with SED fitted stellar population age but an insignificant one with metallicity.