We perform the first quantitative study of the sensitivity of Big Bang Nucleosynthesis to variations in isospin breaking with precise input from lattice QCD calculations. The predicted light nuclear abundances are most sensitive to the neutron-proton mass splitting as both the initial relative abundance of neutrons to protons and the $n \rightleftharpoons p$ weak reaction rates are very sensitive to this quantity. Lattice QCD has been used to determine this mass splitting to greater than 5-sigma, including contributions from both the down-quark up-quark mass splitting, $2\delta = m_d-m_u$ and from electromagnetic coupling of the quarks to the photons with a strength governed by the fine structure constant, $\alpha_{fs}$. At leading order in isospin breaking, the contribution of $\delta$ and $\alpha_{fs}$ to $M_n-M_p$ and the nuclear reaction rates can be varied independently. We use this knowledge and input from lattice QCD to quantitatively study variations of the predicted light nuclear abundances as $\delta$ and $\alpha_{fs}$ are varied. The change in the D and ${}^4$He abundances individually allow for potentially large simultaneous variations in $\delta$ and $\alpha_{fs}$ while maintaining consistency with the observed abundances, however the combined comparison restricts variations in these sources of isospin breaking to less than $\lesssim1.25\%$ at the 3-sigma confidence level. This sensitivity can be used to place tight constraints on prospective beyond the Standard Model theories that would modify these isospin breaking effects in the primordial Universe.