A major goal of the BepiColombo mission to Mercury is the determination of the structure and state of Mercury's interior. Here the BepiColombo rotation experiment has been simulated in order to assess the ability to attain the mission goals and to help lay out a series of constraints on the experiment's possible progress. In the rotation experiment pairs of images of identical surface regions taken at different epochs are used to retrieve information on Mercury's rotation and orientation. The idea is that from observations of the same patch of Mercury's surface at two different solar longitudes of Mercury the orientation of Mercury can be determined, and therefore also the obliquity and rotation variations with respect to the uniform rotation. The estimation of the libration amplitude and obliquity through pattern matching of observed surface landmarks is challenging. The main problem arises from the difficulty to observe the same landmark on the planetary surface repeatedly over the MPO mission lifetime, due to the combination of Mercury's 3:2 spin-orbit resonance, the absence of a drift of the MPO polar orbital plane and the need to combine data from different instruments with their own measurement restrictions. By assuming that Mercury occupies a Cassini state and that the spacecraft operates nominally we show that under worst case assumptions the annual libration amplitude and obliquity can be measured with a precision of respectively 1.4 arcseconds (as) and 1.0 as over the nominal BepiColombo MPO lifetime with about 25 landmarks for rather stringent illumination restrictions. The outcome of the experiment cannot be easily improved by simply relaxing the observational constraints, or increasing the data volume.