The idea that dark matter can be made of intermediate-mass primordial black holes in the $10M_\odot \lesssim M \lesssim 200M_\odot$ range has recently been reconsidered, particularly in the light of the detection of gravitational waves by the LIGO experiment. The existence of even a small fraction of dark matter in black holes should nevertheless result in noticeable quasar gravitational microlensing. Quasar microlensing is sensitive to any type of compact objects in the lens galaxy, to their abundance, and to their mass. We have analyzed optical and X-ray microlensing data from 24 gravitationally lensed quasars to estimate the abundance of compact objects in a very wide range of masses. We conclude that the fraction of mass in black holes or any type of compact objects is negligible outside of the $0.05 M_\odot \lesssim M \lesssim 0.45 M_\odot$ mass range and that it amounts to $20 \pm5$% of the total matter, in agreement with the expected masses and abundances of the stellar component. Consequently, the existence of a significant population of intermediate-mass primordial black holes appears to be inconsistent with current microlensing observations. Therefore, primordial massive black holes are a very unlikely source of the gravitational radiation detected by LIGO.