Antibaryons are produced in our Galaxy in collisions of high energy cosmic rays with the interstellar medium and in old supernova remnants, and possibly, in exotic sources such as primordial black hole evaporation or dark matter annihilations and decays. The search for signals from exotic sources in antiproton data is hampered by large backgrounds from spallation which, within theoretical errors, can solely account for the current data. Due to the higher energy threshold for antideuteron production, which translates into a suppression of the low energy flux from spallations, antideuteron searches have been proposed as a probe for exotic sources. We perform in this paper a comprehensive analysis of the antideuteron fluxes at the Earth expected from known and hypothetical sources in our Galaxy, and we calculate their maximal values consistent with current antiproton data from AMS-02. We find that supernova remnants generate a negligible flux, whereas primordial black hole evaporation and dark matter annihilations or decays may dominate the total flux at low energies. On the other hand, we find that the detection of cosmic antideuterons would require, for the scenarios studied in this paper and assuming optimistic values of the coalescence momentum and solar modulation, an increase of the experimental sensitivity compared to ongoing and planned instruments by at least a factor of 2. Finally, we briefly comment on the prospects for antihelium-3 detection.