The observed scatter in intergalactic Lyman-$\alpha$ opacity at $z \lesssim 6$ requires large-scale fluctuations in the neutral fraction of the intergalactic medium (IGM) after the expected end of reionization. Post-reionization models that explain this scatter invoke fluctuations in either the ionizing ultraviolet background (UVB) or IGM temperature. These models make very different predictions, however, for the relationship between Lyman-$\alpha$ opacity and local density. Here we test these models using Lyman-$\alpha$ emitting galaxies (LAEs) to trace the density field surrounding the longest and most opaque known Lyman-$\alpha$ trough at $z < 6$. Using deep Subaru Hyper Suprime-Cam narrow-band imaging, we find a highly significant deficit of $z \simeq 5.7$ LAEs within 20 Mpc/$h$ of the trough. The results are consistent with a model in which the scatter in Lyman-$\alpha$ opacity near $z \sim 6$ is driven by large-scale UVB fluctuations, and disfavor a scenario in which the scatter is primarily driven by variations in IGM temperature. UVB fluctuations at this epoch present a boundary condition for reionization models, and may help shed light on the nature of the ionizing sources.