Distant BL Lacertae objects emit $\gamma$ rays which interact with the extragalactic background light (EBL), creating electron-positron pairs, and reducing the flux measured by ground-based imaging atmospheric Cherenkov telescopes (IACTs) at very-high energies (VHE). These pairs can Compton-scatter the cosmic microwave background, creating a $\gamma$-ray signature at slightly lower energies observable by the \fermi\ Large Area Telescope (LAT). This signal is strongly dependent on the intergalactic magnetic field (IGMF) strength ($B$) and its coherence length ($L_B$). We use IACT spectra taken from the literature for 5 VHE-detected BL Lac objects, and combine it with LAT spectra for these sources to constrain these IGMF parameters. Low $B$ values can be ruled out by the constraint that the cascade flux cannot exceed that observed by the LAT. High values of $B$ can be ruled out from the constraint that the EBL-deabsorbed IACT spectrum cannot be greater than the LAT spectrum extrapolated into the VHE band, unless the cascade spectrum contributes a sizable fraction of the LAT flux. We rule out low $B$ values ($B< 10^{-19}$ G for $L_B\ge1$\ Mpc) at $>5\sigma$ in all trials with different EBL models and data selection, except when using $>1$ GeV spectra and the lowest EBL models. We were not able to constrain high values of $B$.