Microlensing of stars places significant constraints on sub-planetary-mass compact objects, including primordial black holes, as dark matter candidates. As the lens' Einstein radius in the source plane becomes comparable to the size of the light source, however, source amplification is strongly suppressed, making it challenging to constrain lenses with a mass at or below $10^{-10}$ solar masses, i.e. asteroid-mass objects. Current constraints, using Subaru HSC observations of M31, assume a fixed source size of one solar radius. Here we point out that the actual stars in M31 bright enough to be used for microlensing are typically much larger. We correct the HSC constraints by constructing a source size distribution based on the M31 PHAT survey and on a synthetic stellar catalogue, and by correspondingly weighing the finite-size source effects. We find that the actual HSC constraints are weaker by up to almost three orders of magnitude in some cases, broadening the range of masses for which primordial black holes can be the totality of the cosmological dark matter by almost one order of magnitude.