The processes driving gas accretion on to supermassive black holes (SMBHs) are still poorly understood. Angular momentum conservation prevents gas within $\sim10\,$pc of the black hole from reaching radii $\sim10^{-3}\,$pc where viscous accretion becomes efficient. Here we present simulations of the collapse of a clumpy shell of swept-up isothermal gas, which is assumed to have formed as a result of feedback from a previous episode of AGN activity. The gas falls towards the SMBH forming clumps and streams, which intersect, collide, and often form a disc. These collisions promote partial cancellations of angular momenta, resulting in further infall and more collisions. This continued collisional cascade generates a tail of gas with sufficiently small angular momenta and provides a viable route for gas inflow to sub-parsec scales. The efficiency of this process hardly depends on details, such as gas temperature, initial virial ratio and power spectrum of the gas distribution, as long as it is not strongly rotating. Adding star formation to this picture might explain the near-simultaneous formation of the S-stars (from tidally disrupted binaries formed in plunging gas streams) and the sub-parsec young stellar disc around Sgr A$^{\!\star}$.