We study the survival of substructures (clumps) within larger self-gravitating dark matter halos. Building on scaling relations obtained from N-body calculations of violent relaxation, we argue that the tidal field of galaxies and halos can only destroy substructures if spherical symmetry is imposed at formation. We explore other mechanisms that may tailor the number of halo substructures during the course of virialization. Unless the larger halo is built up from a few large clumps, we find that clump-clump encounters are unlikely to homogenize the halo on a dynamical timescale. Phase mixing would proceed faster in the inner parts and allow for the secular evolution of a stellar disk.