The damping of hot giant dipole resonances is investigated. The contribution of surface scattering is compared with the contribution from interparticle collisions. A unified response function is presented which includes surface damping as well as collisional damping. The surface damping enters the response via the Lyapunov exponent and the collisional damping via the relaxation time. The former is calculated for different shape deformations of quadrupole and octupole type. The surface as well as the collisional contribution each reproduce almost the experimental value, therefore we propose a proper weighting between both contributions related to their relative occurrence due to collision frequencies between particles and of particles with the surface. We find that for low and high temperatures the collisional contribution dominates whereas the surface damping is dominant around the temperatures $\sqrt{3}/2\pi$ of the centroid energy.