We demonstrate how galaxy morphologies can be represented by weighted sums of 'eigengalaxies' and how eigengalaxies can be used in a probabilistic framework to enable principled and simplified approaches in a variety of applications. Eigengalaxies can be derived from a Principal Component Analysis (PCA) of sets of single- or multi-band images. They encode the image space equivalent of basis vectors that can be combined to describe the structural properties of large samples of galaxies in a massively reduced manner. As an illustration, we show how a sample of 10,243 galaxies in the Hubble Space Telescope CANDELS survey, which have visual classifications from Galaxy Zoo, can be represented by just 12 eigengalaxies whilst retaining 96% explained variance. We show how the emphasis of certain eigengalaxy components correspond to visual features (e.g. disc, point source, etc.), and explore the correspondence between combinations of eigengalaxies and features defined in the Galaxy Zoo-CANDELS catalogue. We also describe a probabilistic extension to PCA (PPCA) which enables the eigengalaxy framework to assign probabilities to galaxies and characterise a whole population as a generative distribution. We present four practical applications of the probabilistic eigengalaxy framework that are particularly relevant for the next generation of large imaging surveys: we (i) show how low probability galaxies make for natural candidates for outlier detection (ii) demonstrate how missing data can be predicted (iii) show how a similarity search can be performed on exemplars (iv) demonstrate how unsupervised clustering of objects can be implemented.