The mergers of compact binaries with at least one neutron star component are the potential leading sites of the production and ejection of $r$-process elements. Discoveries of galactic binary pulsars, short gamma-ray bursts, and gravitational-wave detections have all been constraining the rate of these events, while the gravitational wave plus broadband electromagnetic coverage of binary neutron star merger (GW170817) has also placed constraints on the properties (mass and composition) of the merger ejecta. But uncertainties and ambiguities in modeling the optical and infrared emission make it difficult to definitively measure the distribution of heavy isotopes in these mergers. In contrast, gamma rays emitted in the decay of these neutron-rich ejecta may provide a more direct measurement of the yields. We calculate the gamma production in remnants of neutron star mergers, considering two epochs: a kilonova epoch, lasting about two weeks, and a much later epoch of tens and hundreds of thousands of years after the merger. For the kilonova epoch, when the expanding ejecta is still only partially transparent to gamma radiation, we use 3D radiative transport simulations to produce the spectra. We show that the gamma-ray spectra associated with beta- and alpha-decay provide a fingerprint of the ejecta properties and, for a sufficiently nearby remnant, may be detectable, even for old remnants. We compare our gamma spectra with the potential detection limits of next generation detectors, including the Lunar Occultation Explorer (LOX), the All-sky Medium Energy Gamma-ray Observatory (AMEGO), and the Compton Spectrometer and Imager (COSI). We show that fission models can be discriminated via the presence of short-lived fission fragments in the remnant spectra.