Plasma instabilities can be encountered in many branches of physics. This work focuses on relativistic plasmas with applications in theoretical astrophysics and particle accelerator physics. Even though these fields seem to be unrelated the underlying plasma physics processes are often very similar. Two plasma instabilities - the beam-beam instability and the coherent synchrotron radiation instability - are analyzed. The former severely limits the achievable luminosity in storage rings and is related to the two-stream instability which has been proposed as a candidate for the radiation mechanism of radio pulsars. The main emphasis is on coherent synchrotron radiation which can lead to prohibitive energy losses in bunch compressors. Coherent synchrotron radiation also makes up the intense emission of radio waves by pulsars. Simple models based on the linearized Vlasov equation and relativistic magnetohydrodynamics which allow to compute detailed spectra of the emitted radiation are developed.

We investigate the effect of shear in the flow of charged particle equilibria that are unstable to the Coherent Synchrotron Radiation (CSR) instability. Shear may act to quench this instability because it acts to limit the size of the region with a fixed phase relation between emitters. The results are important for the understanding of astrophysical sources of coherent radiation where shear in the flow is likely.

The electromagnetic radiation of a rotating relativistic electron layers is studied numerically using particle-in-cell simulation. The results of the simulation confirm all relevant scaling properties predicted by theoretical models. These models may turn out to be important for the understanding of the coherent synchrotron radiation (CSR) instability that may occur in systems as diverse as particle accelerators radio pulsars.

We study the stability of a collisionless, relativistic, finite-strength, cylindrical layer of charged particles in free space by solving the linearized Vlasov-Maxwell equations and compute the power of the emitted electromagnetic waves. The layer is rotating in an external magnetic field parallel to the layer. This system is of interest to understanding the high brightness temperature of pulsars which cannot be explained by an incoherent radiation mechanism. Coherent synchrotron radiation has also been observed recently in bunch compressors used in particle accelerators. We consider equilibrium layers with a `thermal' energy spread and therefore a non-zero radial thickness. The particles interact with their retarded electromagnetic self-fields. The effect of the betatron oscillations is retained. A short azimuthal wavelength instability is found which causes a modulation of the charge and current densities. The growth rate is found to be an increasing function of the azimuthal wavenumber, a decreasing function of the Lorentz factor, and proportional to the square root of the total number of electrons. We argue that the growth of the unstable perturbation saturates when the trapping frequency of electrons in the wave becomes comparable to the growth rate. Owing to this saturation we can predict the radiation spectrum for a given set of parameters. Our predicted brightness temperatures are proportional to the square of the number of particles and scale by the inverse five-third power of the azimuthal wavenumber which is in rough accord with the observed spectra of radio pulsars.