We simultaneously study the dynamics of the growth of errors and the question of the faithfulness of simulations of $N$-body systems. The errors are quantified through the numerical reversibility of small-$N$ spherical systems, and by comparing fixed-timestep runs with different stepsizes. The errors add randomly, before exponential divergence sets in, with exponentiation rate virtually independent of $N$, but scale saturating as $\sim 1/\sqrt{N}$, in line with theoretical estimates presented. In a third phase, the growth rate is initially driven by multiplicative enhancement of errors, as in the exponential stage. It is then qualitatively different for the phase space variables and mean field conserved quantities (energy and momentum); for the former, the errors grow systematically through phase mixing, for the latter they grow diffusively. For energy, the $N$-variation of the `relaxation time' of error growth follows the $N$-scaling of two-body relaxation. This is also true for angular momentum in the fixed stepsize runs, although the associated error threshold is higher and the relaxation time smaller. Due to shrinking saturation scales, the information loss associated with the exponential instability decreases with $N$ and the dynamical entropy vanishes at any finite resolution as $N \rightarrow \infty$. A distribution function depending on the integrals of motion in the smooth potential is decreasingly affected. In this sense there is convergence to the collisionless limit, despite the persistence of exponential instability on infinitesimal scales. Nevertheless, the slow $N$-variation in its saturation points to the slowness of the convergence.