The quark mass dependences of light element binding energies and nuclear scattering lengths are derived using chiral perturbation theory in combination with non-perturbative methods. In particular, we present new, improved values for the quark mass dependence of meson resonances that enter the nuclear force. A detailed analysis of the theoretical uncertainties arising in this determination is presented. As an application we derive from a comparison of observed and calculated primordial deuterium and helium abundances a stringent limit on the variation of the light quark mass, $\delta m_q/m_q = 0.02 \pm 0.04$. Inclusion of the neutron lifetime modification under the assumption of a variation of the Higgs vacuum expectation value that translates into changing quark, electron, and weak gauge boson masses, leads to a stronger limit, $|\delta m_q/m_q| < 0.009$.