We analyze the impact of Fermi gamma-ray observations (primarily non-detections) of selected nearby galaxies, including dwarf spheroidals, and of clusters of galaxies on decaying dark matter models. We show that the fact that galaxy clusters do not shine in gamma rays puts the most stringent limits available to-date on the lifetime of dark matter particles for a wide range of particle masses and decay final states. In particular, our results put strong constraints on the possibility of ascribing to decaying dark matter both the increasing positron fraction reported by PAMELA and the high-energy feature in the electron-positron spectrum measured by Fermi. Observations of nearby dwarf galaxies and of the Andromeda Galaxy (M31) do not provide as strong limits as those from galaxy clusters, while still improving on previous constraints in some cases.