We put bounds on the variation of the value of the fine structure constant $\alpha$, at the time of Big Bang nucleosynthesis. We study carefully all light elements up to $^7$Li. We correct a previous upper limit on $|\Delta \alpha / \alpha|$ estimated from $^4$He primordial abundance and we find interesting new potential limits (depending on the value of the baryon-to-photon ratio) from $^7$Li, whose production is governed to a large extent by Coulomb barriers. The presently unclear observational situation concerning the primordial abundances preclude a better limit than $|\Delta \alpha/\alpha| \lsim 2\cdot 10^{-2}$, two orders of magnitude less restrictive than previous bounds. In fact, each of the (mutually exclusive) scenarios of standard Big Bang nucleosynthesis proposed, one based on a high value of the measured deuterium primordial abundance and one based on a low value, may describe some aspects of data better if a change in $\alpha$ of this magnitude is assumed.