In a previous paper (Gayon & Bois 2008a), we have shown the general efficiency of retrograde resonances for stabilizing compact planetary systems. Such retrograde resonances can be found when two-planets of a three-body planetary system are both in mean motion resonance and revolve in opposite directions. For a particular two-planet system, we have also obtained a new orbital fit involving such a counter-revolving configuration and consistent with the observational data. In the present paper, we analytically investigate the three-body problem in this particular case of retrograde resonances. We therefore define a new set of canonical variables allowing to express correctly the resonance angles and obtain the Hamiltonian of a system harboring planets revolving in opposite directions. The acquiring of an analytical "rail" may notably contribute to a deeper understanding of our numerical investigation and provides the major structures related to the stability properties. A comparison between our analytical and numerical results is also carried out.

The five planets discovered around the main-sequence star 55 Cnc may represent a case of stable chaos. By using both the Frequency Map Analysis and MEGNO we find that about 15 % of the systems that can be build from the nominal orbital elements of the system are highly chaotic. However, in spite of the fast diffusion rate in the phase space, the planetary system is not destabilized over 400 Myr and close encounters between the planets are avoided.

Most multi-planetary systems are characterized by hot-Jupiters close to their central star, moving on eccentric orbits. From a dynamical point of view, compact multi-planetary systems form a specific class of the general N-body problem (where N>=3). Moreover, extrasolar planets are found in prograde orbits about their host star, and often in mean motion resonances (MMR). In a first step, we study theoretically a new stabilizing mechanism suitable for compact two-planet systems. This mechanism involves counter-revolving orbits forming a retrograde MMR. In a second step, we investigate the feasibility of planetary systems hosting counter-revolving planets. Dynamical stability, observations, and formation processes of these systems are analyzed and discussed. To characterize the dynamical behavior of multi-dimensional planetary systems, we apply our technique of global dynamics analysis based on the MEGNO indicator (Mean Exponential Growth factor of Nearby Orbits) that provides the fine structure of the phase space. In a few cases of possible counter-revolving configurations, we carry out new fits to the observations using the Pikaia genetic algorithm. A statistical study of the stability in the neighborhood of different observed, planetary-systems is completed using a Monte-Carlo method. We analyse the observational data for the HD73526 planetary system and find that counter-revolving configurations may be consistent with the observational data. We highlight the fine and characteristic structure of retrograde MMRs. We demonstrate that retrograde resonances open a family of stabilizing mechanisms involving new apsidal precession behaviors. Considering two possible formation mechanisms (free-floating planet and Slingshot model), we conclude that counter-revolving configurations are feasible.

Multi-planet systems detected until now are in most cases characterized by hot-Jupiters close to their central star as well as high eccentricities. As a consequence, from a dynamical point of view, compact multi-planetary systems form a variety of the general N-body problem (with N >= 3), whose solutions are not necessarily known. Extrasolar planets are up to now found in prograde (i.e. direct) orbital motions about their host star and often in mean-motion resonances (MMR). In the present paper, we investigate a theoretical alternative suitable for the stability of compact multi-planetary systems. When the outer planet moves on a retrograde orbit in MMR with respect to the inner planet, we find that the so-called retrograde resonances present fine and characteristic structures particularly relevant for dynamical stability. We show that retrograde resonances and their resources open a family of stabilizing mechanisms involving specific behaviors of apsidal precessions. We also point up that for particular orbital data, retrograde MMRs may provide more robust stability compared to the corresponding prograde MMRs.