Mercury is entrapped in a 3:2 resonance: it rotates on its axis three times for every two revolutions it makes around the Sun. It is generally accepted that this is due to the large value of the eccentricity of its orbit. However, the mathematical model originally introduced to study its spin-orbit evolution proved not to be entirely convincing, because of the expression commonly used for the tidal torque. Only recently, in a series of papers mainly by Efroimsky and Makarov, a different model for the tidal torque has been proposed, which has the advantages of being more realistic, and of providing a higher probability of capture in the 3:2 resonance with respect to the previous models. On the other hand, a drawback of the model is that the function describing the tidal torque is not smooth and consists of a superposition of kinks, so that both analytical and numerical computations turn out to be rather delicate: indeed, standard perturbation theory based on power series expansion cannot be applied and the implementation of a fast algorithm to integrate the equations of motion numerically requires a high degree of care. In this paper, we make a detailed study of the spin-orbit dynamics of Mercury, as predicted by the realistic model: In particular, we present numerical and analytical results about the nature of the librations of Mercury's spin in the 3:2 resonance. The results provide evidence that the librations are quasi-periodic in time.