The origin of cycle-to-cycle variations in solar activity is currently the focus of much interest. It has recently been pointed out that large individual active regions with atypical properties can have a significant impact on the long term behaviour of solar activity. We investigate this possibility in more detail using a recently developed 2$\times$2D dynamo model of the solar magnetic cycle. We find that even a single "rogue" bipolar magnetic region (BMR) in the simulations can have a major effect on the further development of solar activity cycles, boosting or suppressing the amplitude of subsequent cycles. In extreme cases an individual BMR can completely halt the dynamo, triggering a grand minimum. Rogue BMRs also have the potential to induce significant hemispheric asymmetries in the solar cycle. To study the effect of rogue BMRs in a more systematic manner, a series of dynamo simulations were conducted, in which a large test BMR was manually introduced in the model at various phases of cycles of different amplitudes. BMRs emerging in the rising phase of a cycle can modify the amplitude of the ongoing cycle while BMRs emerging in later phases will only impact subsequent cycles. In this model, the strongest impact on the subsequent cycle occurs when the rogue BMR emerges around cycle maximum at low latitudes but the BMR does not need to be strictly cross-equatorial. Active regions emerging as far as $20^{\circ}$ from the equator can still have a significant impact. We demonstrate that the combined effect of the magnetic flux, tilt angle and polarity separation of the BMR on the dynamo is via their contribution to the dipole moment, $\delta D_{\mathrm{BMR}}$. Our results indicate that prediction of the amplitude, starting epoch and duration of a cycle requires an accurate accounting of a broad range of active regions emerging in the previous cycle.