Binary black holes are the primary endpoint of massive stellar evolution. Their properties provide a unique opportunity to constrain binary evolution, which is still poorly understood. In this paper, we predict the inventory of binary black holes and their merger products in/around the Milky Way, and detail their main properties. We present the first combination of a high-resolution cosmological simulation of a Milky Way-mass galaxy with a binary population synthesis model. The hydrodynamic simulation, taken from the FIRE project, provides a cosmologically realistic star formation history for the galaxy and its stellar halo and satellites. We apply a metallicity-dependent evolutionary model to the star particles to produce individual binary black holes. We find that a million binary black holes have merged in the model Milky Way, and 3 million binaries are still present, with an average mass of 28 Msun per binary. Because the black hole progenitors are biased towards low metallicity stars, half reside in the stellar halo and satellites and 40 per cent of the binaries were formed outside the main galaxy. This trend increases with the masses of the black holes. The numbers and mass distribution of the merged systems is compatible with the LIGO/Virgo detections. Observations of these black holes will be challenging, both with electromagnetic methods and LISA. We find that a cosmologically realistic star formation history, with self-consistent metal enrichment and Galactic accretion history, are key ingredients for determining binary black hole rates that can be compared with observations to constrain massive binary evolution.