We study the Fock description of a quantum free field on the three-sphere with a mass that depends explicitly on time, also interpretable as an explicitly time dependent quadratic potential. We show that, under quite mild restrictions on the time dependence of the mass, the specific Fock representation of the canonical commutation relations which is naturally associated with a massless free field provides a unitary dynamics even when the time varying mass is present. Moreover, we demonstrate that this Fock representation is the only acceptable one, up to unitary equivalence, if the vacuum has to be SO(4)-invariant (i.e., invariant under the symmetries of the field equation) and the dynamics is required to be unitary. In particular, the analysis and uniqueness of the quantization can be applied to the treatment of cosmological perturbations around Friedmann-Robertson-Walker spacetimes with the spatial topology of the three-sphere, like e.g. for gravitational waves (tensor perturbations). In addition, we analyze the extension of our results to free fields with a time dependent mass defined on other compact spatial manifolds. We prove the uniqueness of the Fock representation in the case of a two-sphere as well, and discuss the case of a three-torus.