Rossby waves (r-modes) in rapidly rotating neutron stars are unstable because of the emission of gravitational radiation. We study saturation of this instability by nonlinear transfer of energy to stellar "inertial" oscillation modes. We present detailed calculations of stellar inertial modes in the WKB limit, their linear damping by bulk and shear viscosity, and the nonlinear coupling forces among these modes. The saturation amplitude is derived in the extreme limits of strong or weak driving by radiation reaction, as compared to the damping rate of low order inertial modes. We find the saturation energy is {\it extremely small}, at least four orders of magnitude smaller than that found by previous investigators. We discuss the consequences of this result for spin evolution of young neutron stars, and neutron stars being spun up by accretion in Low Mass X-ray Binaries.We also discuss the detection of these gravitational waves by LIGO.

We develop the formalism required to study the nonlinear interaction of modes in rotating Newtonian stars in the weakly nonlinear regime. The formalism simplifies and extends previous treatments. At linear order, we elucidate and extend slightly a formalism due to Schutz, show how to decompose a general motion of a rotating star into a sum over modes, and obtain uncoupled equations of motion for the mode amplitudes under the influence of an external force. Nonlinear effects are added perturbatively via three-mode couplings. We describe a new, efficient way to compute the coupling coefficients, to zeroth order in the stellar rotation rate, using spin-weighted spherical harmonics. We apply this formalism to derive some properties of the coupling coefficients relevant to the nonlinear interactions of unstable r-modes in neutron stars, postponing numerical integrations of the coupled equations of motion to a later paper. From an astrophysical viewpoint, the most interesting result of this paper is that many couplings of r-modes to other rotational modes (modes with zero frequencies in the non-rotating limit) are small: either they vanish altogether because of various selection rules, or they vanish to lowest order in the angular velocity. In zero-buoyancy stars, the coupling of three r-modes is forbidden entirely and the coupling of two r-modes to one hybrid rotational mode vanishes to zeroth order in rotation frequency. In incompressible stars, the coupling of any three rotational modes vanishes to zeroth order in rotation frequency.