A large number of studies have responded to the growing body of confirmed terrestrial habitable zone exoplanets by presenting models of various possible climates. However, the impact of the partial pressure of background gases such as N$_2$ has been poorly-explored, despite the abundance of N$_2$ in Earth's atmosphere and the lack of constraints on its typical abundance in terrestrial planet atmospheres. We use PlaSim, a fast 3D climate model, to simulate many hundreds of climates with varying N$_2$ partial pressures, insolations, and surface characteristics to identify the impact of the background gas partial pressure on the climate. We find that the climate's response is nonlinear and highly sensitive to the background gas partial pressure. We identify pressure broadening of CO$_2$ and H$_2$O absorption lines, amplification of warming or cooling by the water vapor greenhouse positive feedback, heat transport efficiency, and cooling through Rayleigh scattering as the dominant competing mechanisms that determine the equilibrium climate for a given N$_2$ partial pressure. Finally, we show that different amounts of N$_2$ should have a significant effect on broadband reflected light observations of terrestrial exoplanets.