Ultralight scalar dark matter can interact with all massive Standard Model particles through a universal coupling. Such a coupling modifies the Standard Model particle masses and affects the dynamics of Big Bang Nucleosynthesis. We model the cosmological evolution of the dark matter, taking into account the modifications of the scalar mass by the environment as well as the full dynamics of Big Bang Nucleosynthesis. We find that precision measurements of the helium-4 abundance set stringent constraints on the available parameter space, and that these constraints are strongly affected by both the dark matter environmental mass and the dynamics of the neutron freeze-out. Furthermore, we perform the analysis in both the Einstein and Jordan frames, the latter of which allows us to implement the model into numerical Big Bang Nucleosynthesis codes and analyze additional light elements. The numerical analysis shows that the constraint from helium-4 dominates over deuterium, and that the effect on lithium is insufficient to solve the lithium problem. Comparing to several other probes, we find that Big Bang Nucleosynthesis sets the strongest constraints for the majority of the parameter space.