The theory for single stellar evolution predicts a gap in the mass distribution of black holes (BHs) between approximately 45-130M$_{\odot}$, the so-called "pair-instability mass gap". We examine whether BHs can pollute the gap after accreting from a stellar companion. To this end, we simulate the evolution of isolated binaries using a population synthesis code, where we allow for super-Eddington accretion. Under our most extreme assumptions, we find that at most about 2% of all merging binary BH systems contains a BH with a mass in the pair-instability mass gap, and we find that less than 0.5% of the merging systems has a total mass larger than 90M$_{\odot}$. We find no merging binary BH systems with a total mass exceeding 100M$_{\odot}$. We compare our results to predictions from several dynamical pathways to pair-instability mass gap events and discuss the distinguishable features. We conclude that the classical isolated binary formation scenario will not significantly contribute to the pollution of the pair-instability mass gap. The robustness of the predicted mass gap for the isolated binary channel is promising for the prospective of placing constraints on (i) the relative contribution of different formation channels, (ii) the physics of the progenitors including nuclear reaction rates, and (iii), tentatively, the Hubble parameter.