Terrestrial planets are believed to be formed via giant impacts of Mars-sized protoplanets. Planets formed via giant impacts have highly eccentric orbits. A swarm of planetesimals around the planets may lead to eccentricity damping for the planets via the equipartition of random energies (dynamical friction). However, dynamical friction increases eccentricities of planetesimals, resulting in high velocity collisions between planetesimals. The collisional cascade grinds planetesimals to dust until dust grains are blown out due to radiation pressure. Therefore, the total mass of planetesimals decreases due to collisional fragmentation, which weakens dynamical friction. We investigate the orbital evolution of protoplanets in a planetesimal disk, taking into account collisional fragmentation of planetesimals. For 100 km-sized or smaller planetesimals, dynamical friction is insignificant for eccentricity damping of planets because of collisional fragmentation. On the other hand, giant impacts eject collisional fragments. Although the total mass of giant impact ejecta is 0.1-0.3 Earth masses, the largest impact ejecta are ~ 1,000 km in size. We also investigate the orbital evolution of single planets with initial eccentricities 0.1 in a swarm of such giant impact ejecta. Although the total mass of giant impact ejecta decreases by a factor of 3 in 30 Myrs, eccentricities of planets are damped down to the Earth level (~0.01) due to interaction with giant impact ejecta. Therefore, giant impact ejecta play an important role for determination of terrestrial planet orbits.