Statistical inference more often than not involves models which are non-linear in the parameters thus leading to non-Gaussian posteriors. Many computational and analytical tools exist that can deal with non-Gaussian distributions, and empirical Gaussianisation transforms can reduce the amount of non-Gaussianity in a distribution. Alternatively, in this work, we employ methods from information geometry. The latter formulates a set of probability distributions for some given model as a manifold employing a Riemannian structure, equipped with a metric, the Fisher information. In this framework we study the differential geometrical meaning of non-Gaussianities in a higher order Fisher approximation, and their respective transformation behaviour under re-parameterisation, which corresponds to a chart transition on the statistical manifold. While weak non-Gaussianities vanish in normal coordinates in a first order approximation, one can in general not find transformations that discard non-Gaussianities globally. As an application we consider the likelihood of the supernovae distance-redshift relation in cosmology for the parameter pair ($\Omega_{\mathrm{m_0}}$, $w$). We demonstrate the connection between confidence intervals and geodesic length and demonstrate how the Lie-derivative along the degeneracy directions gives hints at possible isometries of the Fisher metric.