We present two-dimensional inviscid hydrodynamic simulations of a protoplanetary disk with an embedded planet, emphasizing the evolution of potential vorticity (the ratio of vorticity to density) and its dependence on numerical resolutions. By analyzing the structure of spiral shocks made by the planet, we show that progressive changes of the potential vorticity caused by spiral shocks ultimately lead to the excitation of a secondary instability. We also demonstrate that very high numerical resolution is required to both follow the potential vorticity changes and identify the location where the secondary instability is first excited. Low-resolution results are shown to give the wrong location. We establish the robustness of a secondary instability and its impact on the torque onto the planet. After the saturation of the instability, the disk shows large-scale non-axisymmetry, causing the torque on the planet to oscillate with large amplitude. The impact of the oscillating torque on the protoplanet's migration remains to be investigated.

(abridged) We study the nonlinear evolution of the Rossby wave instability in thin disks using global 2D hydrodynamic simulations. The key questions we are addressing in this paper are: (1) What happens when the instability becomes nonlinear? Specifically, does it lead to vortex formation? (2) What is the detailed behavior of a vortex? (3) Can the instability sustain itself and can the vortex last a long time? Among various initial equilibria that we have examined, we generally find that there are three stages of the disk evolution: (1) The exponential growth of the initial small amplitude perturbations. This is in excellent agreement with the linear theory; (2) The production of large scale vortices and their interactions with the background flow, including shocks. Significant accretion is observed due to these vortices. (3) The coupling of Rossby waves/vortices with global spiral waves, which facilitates further accretion throughout the whole disk. Even after more than 20 revolutions at the radius of vortices, we find that the disk maintains a state that is populated with vortices, shocks, spiral waves/shocks, all of which transport angular momentum outwards. We elucidate the physics at each stage and show that there is an efficient outward angular momentum transport in stages (2) and (3) over most parts of the disk, with an $\alpha$ in the range from $10^{-4}$ to $10^{-2}$. We show why such vortices prove to be almost ideal ``units'' in transporting angular momentum outwards, namely by positively correlating the radial and azimuthal velocity components. We find some special cases in which entropy can remain the same while angular momentum is transported. We conclude that Rossby wave/vortex instability is an efficient, purely hydrodynamic mechanism for angular momentum transport in thin disks.