Galaxy-galaxy strong gravitational lenses have become a popular probe of dark matter (DM) by providing a window into structure formation on the smallest scales. In particular, the convergence power spectrum of subhalos within lensing galaxies has been suggested as a promising observable to study DM. However, the distances involved in strong-lensing systems are vast, and we expect the relevant volume to contain line-of-sight (LOS) halos that are not associated with the main lens. We develop a formalism to calculate the effect of LOS halos as an effective convergence power spectrum. The multi-lens plane equation couples the angular deflections of consecutive lens planes, but by assuming that the perturbations due to the LOS halos are small, we show that they can be projected onto the main-lens plane as effective subhalos. We test our formalism by simulating lensing systems using the full multi-plane lens equation and find excellent agreement. We show how the relative contribution of LOS halos and subhalos depends on the source and lens redshift, as well as the assumed halo and subhalo mass functions. For a fiducial system with fraction of DM halo mass in substructure $f_{\rm sub}=0.4\%$ for subhalo masses $[10^5-10^8]\rm{M}_{\odot}$, the interloper contribution to the power spectrum is at least several times greater than that of subhalos for source redshifts $z_s\gtrsim0.5$. Furthermore, it is likely that for the SLACS and BELLS lenses the interloper contribution dominates: $f_{\rm sub}\gtrsim2\%$ ($4\%$) is needed for subhalos to dominate in SLACS (BELLS), which is higher than current upper bounds on $f_{\rm sub}$ for our mass range. Since the halo mass function is better understood from first principles, the dominance of interlopers in galaxy-galaxy lenses with high-quality imaging can be seen as a significant advantage when translating this observable into a constraint on DM.