Mercury is the target of two space missions: MESSENGER (NASA) which orbit insertion is planned for March 2011, and ESA/JAXA BepiColombo, that should be launched in 2014. Their instruments will observe the surface of the planet with a high accuracy (about 1 arcsec for BepiColombo), what motivates studying its rotation. Mercury is assumed to be composed of a rigid mantle and an at least partially molten core. We here study the influence of the core-mantle interactions on the rotation perturbed by the solar gravitational interaction, by modeling the core as an ellipsoidal cavity filled with inviscid fluid of constant uniform density and vorticity. We use both analytical (Lie transforms) and numerical tools to study this rotation, with different shapes of the core. We express in particular the proper frequencies of the system, because they characterize the response of Mercury to the different solicitations, due to the orbital motion of Mercury around the Sun. We show that, contrary to its size, the shape of the core cannot be determined from observations of either longitudinal or polar motions. However, we highlight the strong influence of a resonance between the proper frequency of the core and the spin of Mercury that raises the velocity field inside the core. We show that the key parameter is the polar flattening of the core. This effect cannot be directly derived from observations of the surface of Mercury, but we cannot exclude the possibility of an indirect detection by measuring the magnetic field.