The subject of this paper is a quantification of the information content of cosmological probes of the large-scale structures, specifically of temperature and polarisation anisotropies in the cosmic microwave background, CMB-lensing, weak cosmic shear and galaxy clustering, in terms of Information theory measures like information entropies. We aim to establish relationships for Gaussian likelihoods, between conventional measures of statistical uncertainties and information entropies. Furthermore, we extend these studies to the computation of (Bayesian) evidences and the power of measurement to distinguish between competing models. We investigate in detail how cosmological data decreases information entropy by reducing statistical errors and by breaking degeneracies. In addition, we work out how tensions between data sets increase information entropy and quantify this effect in three examples: the discrepancy in $\Omega_m$ and $\sigma_8$ between the CMB and weak lensing, the role of intrinsic alignments in weak lensing data when attempting the dark energy equation of state parameters, and the famous $H_0$-tension between Cepheids in the Hubble keystone project and the cosmic microwave background as observed by Planck.