The streaming instability and Kelvin--Helmholtz instability are considered the two major sources causing clumping of dust particles and turbulence in the dust layer of a protoplanetary disk as long as we consider the dead zone where the magneto-rotational instability does not grow. Extensive numerical simulations have been carried out in order to elucidate the condition for the development of particle clumping caused by the streaming instability. In this paper, a set of two parameters suitable for classifying the numerical results is proposed. One is the Stokes number that has been employed in previous works and the other is the dust particle column density that is nondimensionalized using the gas density in the midplane, Keplerian angular velocity, and difference between the Keplerian and gaseous orbital velocities. The magnitude of dust clumping is a measure of the behavior of the dust layer. Using three-dimensional numerical simulations of dust particles and gas based on Athena code v. 4.2, it is confirmed that the magnitude of dust clumping for two disk models are similar if the corresponding sets of values of the two parameters are identical to each other, even if the values of the metallicity (i.e., the ratio of the columns density of the dust particles to that of the gas) are different.

Instabilities of the dust layer in a protoplanetary disk are investigated. It is known that the streaming instability develops and dust density concentration occurs in a situation where the initial dust density is uniform. This work considers the effect of initial dust density gradient vertical to the midplane. Dust and gas are treated as different fluids. Pressure of dust fluid is assumed to be zero. The gas friction time is assumed to be constant. Axisymmetric two-dimensional numerical simulation was performed using the spectral method. We found that an instability develops with a growth rate on the order of the Keplerian angular velocity even if the gas friction time multiplied by the Keplerian angular velocity is as small as 0.001. This instability is powered by two sources: (1) the vertical shear of the azimuthal velocity, and (2) the relative motion of dust and gas coupled with the dust density fluctuation due to advection. This instability diffuses dust by turbulent advection and the maximum dust density decreases. This means that the dust concentration by the streaming instability which is seen in the case of a uniform initial dust density becomes ineffective as dust density gradient increases by the dust settling toward the midplane.

Nonlinear numerical simulations are performed to investigate the density evolution in the dust layer of a protoplanetary disk due to the gravitational instability and dust settling toward the midplane. We assume the region where the radial pressure gradient at equilibrium is negligible so that the shear-induced instability is avoided, and also restrict to an axisymmetric perturbation as a first step of nonlinear numerical simulations of the gravitational instability. We find that there are two different evolutional paths of the gravitational instability depending on the nondimensional gas friction time, which is defined as the product of the gas friction time and the Keplerian angular velocity. If the nondimensional gas friction time is equal to 0.01, the gravitational instability grows faster than dust settling. On the other hand, if the nondimensional gas friction time is equal to 0.1, dust aggregates settle sufficiently before the gravitational instability grows. In the latter case, an approximate analytical calculation reveals that dust settling is faster than the growth of the gravitational instability regardless of the dust density at the midplane. Thus, the dust layer becomes extremely thin and may reach a few tenth of the material density of the dust before the gravitational instability grows.

We show that the planetesimal formation due to the gravitational fragmentation of a dust layer in a protoplanetary disk is possible. The dust density distribution in the dust layer would approach the constant Richardson number distribution due to the dust stirring by the shear instability and dust settling. We perform the analysis of the shear instability of dust layer in a protoplanetary disk with the constant Richardson number density distribution. Our study revealed that this distribution is stable against the shear instability even if the dust density at the midplane reaches the critical density of the gravitational instability, and the planetesimal formation through the gravitational fragmentation of the dust layer can occur even for the dust to gas surface density ratio with the solar composition.

The linear analysis of the instability due to vertical shear in the dust layer of the solar nebula is performed. The following assumptions are adopted throughout this paper: (1) The self-gravity of the dust layer is neglected. (2) One fluid model is adopted, where the dust aggregates have the same velocity with the gas due to strong coupling by the drag force. (3) The gas is incompressible. The calculations with both the Coriolis and the tidal forces show that the tidal force has a stabilizing effect. The tidal force causes the radial shear in the disk. This radial shear changes the wave number of the mode which is at first unstable, and the mode is eventually stabilized. Thus the behavior of the mode is divided into two stages: (1) the first growth of the unstable mode which is similar to the results without the tidal force, and (2) the subsequent stabilization due to an increase of the wave number by the radial shear. If the midplane dust/gas density ratio is smaller than 2, the stabilization occurs before the unstable mode grows largely. On the other hand, the mode grows faster by one hundred orders of magnitude, if this ratio is larger than 20. Because the critical density of the gravitational instability is a few hundreds times as large as the gas density, the hydrodynamic instability investigated in this paper grows largely before the onset of the gravitational instability. It is expected that the hydrodynamic instability develops turbulence in the dust layer and the dust aggregates are stirred up to prevent from settling further. The formation of planetesimals through the gravitational instabilities is difficult to occur as long as the dust/gas surface density ratio is equal to that for the solar abundance.