In flavored dark matter models, dark matter can scatter off of nuclei through Higgs and photon exchange, both of which can arise from renormalizable interactions and individually lead to strong constraints from direct detection. While these two interaction channels can destructively interfere in the scattering amplitude, for a thermal relic with equal abundances for the dark matter particle and its antiparticle, this produces no effect on the total event rate. Focusing on lepton-flavored dark matter, we show that it is quite natural for dark matter to have become asymmetric during high-scale leptogenesis, and that in this case the direct detection bounds can be significantly weakened due to interference. We quantify this by mapping out and comparing the regions of parameter space that are excluded by direct detection for the symmetric and asymmetric cases of lepton-flavored dark matter. In particular, we show that the entire parameter region except for a narrow Higgs resonance window is ruled out in the symmetric case for fermion dark matter when the coupling to the Higgs dominates over the coupling to leptons, while large portions of parameter space are still allowed for the asymmetric case. The same is also true for a dark matter mass above 8 GeV for scalar dark matter when the coupling to leptons dominates over the coupling to the Higgs.