Super-massive black holes (SMBHs) observed to have masses of $M_\bullet \sim 10^9 \, \mathrm{M_\odot}$ at $z\gtrsim6$, $<1$ Gyr after the Big Bang, are thought to have been seeded by massive black holes which formed before growing concurrently with the formation of their host galaxies. We model analytically the idealised growth of seed black holes, fed through gas inflow from growing proto-galaxy discs. The inflow depends on the disc gravitational stability and thus varies with black hole and disc mass. We find that for a typical host halo, the efficiency of angular momentum transport, as parametrised by the disc viscosity, is the limiting factor in determining the inflow rate and the black hole accretion rate. For our fiducial case we find an upper black hole mass estimate of $M_\bullet \sim 1.8 \times 10^7 \, \mathrm{M_{\odot}}$ at $z=6$. Only in the extreme case of $\sim 10^{16}$ M$_{\odot}$ haloes at $z=6$ produces SMBH masses of $\sim 10^9$ M$_{\odot}$. However, the number density of such haloes is many orders of magnitude below the estimated 1 Gpc$^{-3}$ of SMBHs at $z=6$, indicating that viscosity driven accretion is too inefficient to feed the growth of seeds into $M_\bullet \sim 10^9 \, \mathrm{M_\odot}$ SMBHs by $z \sim 6$. We demonstrate that major mergers are capable of resolving the apparent discrepancy in black hole mass at $z=6$, with some dependence on the exact choice of orbital parameters of the merger.