During and after recombination, in addition to Thomson scattering with free electrons, photons also coupled to neutral hydrogen and helium atoms through Rayleigh scattering. This coupling influences both CMB anisotropies and the distribution of matter in the Universe. The frequency-dependence of the Rayleigh cross section breaks the thermal nature of CMB temperature and polarization anisotropies and effectively doubles the number of variables needed to describe CMB intensity and polarization statistics, while the additional atomic coupling changes the matter distribution and the lensing of the CMB. We introduce a new method to capture the effects of Rayleigh scattering on cosmological power spectra. Rayleigh scattering modifies CMB temperature and polarization anisotropies at the $\sim\!1 \%$ level at $353 {\rm GHz}$ (scaling $\propto \nu^4$), and modifies matter correlations by as much as $\sim\!0.3\%$. We show the Rayleigh signal, especially the cross-spectra between the thermal (Rayleigh) E-polarization and Rayleigh (thermal) intensity signal, may be detectable with future CMB missions even in the presence of foregrounds, and how this new information might help to better constrain the cosmological parameters.