We compare theoretical dust yields for stars with mass 1 Msun < mstar < 8 Msun, and metallicities 0.001 < Z < 0.008 with observed dust production rates (DPR) by carbon- rich and oxygen-rich Asymptotic Giant Branch (C-AGB and O-AGB) stars in the Large and Small Magellanic Clouds (LMC, SMC). The measured DPR of C-AGB in the LMC are reproduced only if the mass loss from AGB stars is very efficient during the carbon-star stage. The same yields over-predict the observed DPR in the SMC, suggesting a stronger metallicity dependence of the mass-loss rates during the carbon- star stage. DPR of O-AGB stars suggest that rapid silicate dust enrichment occurs due to efficient hot-bottom-burning if mstar > 3 Msun and Z > 0.001. When compared to the most recent observations, our models support a stellar origin for the existing dust mass, if no significant destruction in the ISM occurs, with a contribution from AGB stars of 70% in the LMC and 15% in the SMC.