Using the canonical Hamilton-Jacobi approach we study the Lynden-Bell (1979) concept of bar formation based on the idea of orbital trapping parallel to the long or short axes of the oval potential distortion. The concept considered a single parameter -- a sign of the derivative of the precession rate over angular momentum, determining the orientation of the trapped orbits. We derived a perturbation Hamiltonian which includes two more parameters characterising the background disc and the perturbation, that just as important as the previously known. This allows us to link this old theory with the matrix approach in linear perturbation theory, the theory of weak bars, and explain some features of the nonlinear secular evolution observed in N-body simulations.

A new so-called `gravitational loss-cone instability' in stellar systems has recently been investigated theoretically in the framework of linear perturbation theory and proved to be potentially important in understanding the physical processes in centres of galaxies, star clusters, and the Oort comet cloud. Using N-body simulations of a toy model, we confirm previous findings for the harmonic dominating potential and go beyond the linear theory. Unlike the well-known instabilities, the new one shows no notable change in the spherical geometry of the cluster, but it significantly accelerates the speed of diffusion of particles in phase space leading to an early instability saturation.

A new so-called `gravitational loss-cone instability' in stellar systems has recently been investigated theoretically in the framework of linear perturbation theory and proved to be potentially important in understanding the physical processes in centres of galaxies, star clusters, and the Oort comet cloud. Using N-body simulations, we confirm previous findings and go beyond the linear theory. Unlike the well-known instabilities, the new one shows no notable change in spherical geometry of the cluster, but it significantly accelerates the speed of diffusion of particles in phase space leading to a repopulation of the loss cone and early instability saturation.

Interaction of spiral density waves with stars in the vicinity of the inner Lindblad resonance in galactic discs is investigated using the linear perturbation theory and the leading orders in the epicyclic and WKB approximations. In analogy with shear flows in hydrodynamics, we conjecture that a weak nonlinearity in a narrow resonance region modifies the standard (Landau-Lin) bypass rule of the singularity to the integration in the principal value sense. This indeed leads to the reflection of the spiral wave instead of absorption, but the detailed picture looks awkward: the intervals of the wave weakening alternate with the intervals of the wave growth, so that the net absorption is absent. Incidentally, we rectify the result concerning leading spiral waves obtained earlier for the standard bypass rule.

Stationary stellar systems with radially elongated orbits are subject to radial orbit instability -- an important phenomenon that structures galaxies. Antonov (1973) presented a formal proof of the instability for spherical systems in the limit of purely radial orbits. However, such spheres have highly inhomogeneous density distributions with singularity $\sim 1/r^2$, resulting in an inconsistency in the proof. The proof can be refined, if one considers an orbital distribution close to purely radial, but not entirely radial, which allows to avoid the central singularity. For this purpose we employ non-singular analogs of generalised polytropes elaborated recently in our work in order to derive and solve new integral equations adopted for calculation of unstable eigenmodes in systems with nearly radial orbits. In addition, we establish a link between our and Antonov's approaches and uncover the meaning of infinite entities in the purely radial case. Maximum growth rates tend to infinity as the system becomes more and more radially anisotropic. The instability takes place both for even and odd spherical harmonics, with all unstable modes developing rapidly, i.e. having eigenfrequencies comparable to or greater than typical orbital frequencies. This invalidates orbital approximation in the case of systems with all orbits very close to purely radial.

We consider a two-parametric family of radially anisotropic models with non-singular density distribution in the centre. If highly eccentric orbits are locked near the centre, the characteristic growth rate of the instability is much less than the Jeans and dynamic frequencies of the stars (slow modes). The instability occurs only for even spherical harmonics and the perturbations are purely growing (aperiodic). On the contrary, if all orbits nearly reach the outer radius of the sphere, both even and odd harmonics are unstable. Unstable odd modes oscillate having characteristic frequencies of the order of the dynamical frequencies (fast modes). Unstable even harmonics contain a single aperiodic mode and several oscillatory modes, the aperiodic mode being the most unstable. The question of the nature of the radial orbit instability (ROI) is revisited. Two main interpretations of ROI were suggested in the literature. The first one refers to the classical Jeans instability associated with the lack of velocity dispersion of stars in the transverse direction. The second one refers to Lynden-Bell's orbital approach to bar formation in disc galaxies, which implies slowness and bi-symmetry of the perturbation. Oscillatory modes, odd spherical harmonics modes, and non-slow modes found in one of the models show that the orbital interpretation is not the only possible.

We study a new class of equilibrium two-parametric distribution functions of spherical stellar systems with radially anisotropic velocity distribution of stars. The models are less singular counterparts of the so called generalized polytropes, widely used in works on equilibrium and stability of gravitating systems in the past. The offered models, unlike the generalized polytropes, have finite density and potential in the center. The absence of the singularity is necessary for proper consideration of the radial orbit instability, which is the most important instability in spherical stellar systems. Comparison of the main observed parameters (potential, density, anisotropy) predicted by the present models and other popular equilibrium models is provided.

We discuss some contradictions found in the literature concerning the problem of stability of collisionless spherical stellar systems which are the simplest anisotropic generalization of the well-known polytrope models. Their distribution function $F(E,L)$ is a product of power-low functions of the energy $E$ and the angular momentum $L$, i.e. $F\propto L^{-s}(-E)^q$. On the one hand, calculation of the growth rates in the framework of linear stability theory and N-body simulations show that these systems become stable when the parameter $s$ characterizing the velocity anisotropy of the stellar distribution is lower than some finite threshold value, $s<s_\textrm{crit}$. On the other hand Palmer & Papaloizou (1987) showed that the instability remained up to the isotropic limit $s=0$. Using our method of determining the eigenmodes for stellar systems, we show that the growth rates in weakly radially-anisotropic systems are indeed positive, but decrease exponentially as the parameter $s$ approaches zero, i.e. $\gamma\propto \exp(-s_{\ast}/s)$. In fact, for the systems with finite lifetime this means stability.

Small perturbations in spherical and thin disk stellar clusters surrounding massive a black hole are studied. Due to the black hole, stars with sufficiently low angular momentum escape from the system through the loss cone. We show that stability properties of spherical clusters crucially depend on whether the distribution of stars is monotonic or non-monotonic in angular momentum. It turns out that only non-monotonic distributions can be unstable. At the same time the instability in disk clusters is possible for both types of distributions.

We study spherical and disk clusters in a near-Keplerian potential of galactic centers or massive black holes. In such a potential orbit precession is commonly retrograde, i.e. direction of the orbit precession is opposite to the orbital motion. It is assumed that stellar systems consist of nearly radial orbits. We show that if there is a loss cone at low angular momentum (e.g., due to consumption of stars by a black hole), an instability similar to loss-cone instability in plasma may occur. The gravitational loss-cone instability is expected to enhance black hole feeding rates. For spherical systems, the instability is possible for the number of spherical harmonics $l \ge 3$. If there is some amount of counter-rotating stars in flattened systems, they generally exhibit the instability independently of azimuthal number $m$. The results are compared with those obtained recently by Tremaine for distribution functions monotonically increasing with angular momentum. The analysis is based on simple characteristic equations describing small perturbations in a disk or a sphere of stellar orbits highly elongated in radius. These characteristic equations are derived from the linearized Vlasov equations (combining the collisionless Boltzmann kinetic equation and the Poisson equation), using the action-angle variables. We use two techniques for analyzing the characteristic equations: the first one is based on preliminary finding of neutral modes, and the second one employs a counterpart of the plasma Penrose-Nyquist criterion for disk and spherical gravitational systems.