We consider whether stellar collisions can explain the observed depletion of red giants in the Galactic center. We model the stellar population with two different IMFs: 1) the Miller-Scalo and 2) a much flatter IMF. In the former case, low-mass main-sequence stars dominate the population, and collisions are unable to remove red giants out to 0.4 pc although brighter red giants much closer in may be depleted via collisions with stellar-mass black holes. For a much flatter IMF, the stellar population is dominated by compact remnants (i.e. black holes, white dwarfs and neutron stars). The most common collisions are then those between main-sequence stars and compact remnants. Such encounters are likely to destroy the main-sequence stars and thus prevent their evolution into red giants. In this way, the red-giant population could be depleted out to 0.4 pc matching observations. If this is the case, it implies the Galactic center contains a much larger population of stellar-mass black holes than would be expected from a regular IMF. This may in turn have implications for the formation and growth of the central supermassive black hole.