Broad infrared emission features (e.g., at 3.3, 6.2, 7.7, 8.6, and 11.3 microns) from the gas phase interstellar medium have long been attributed to polycyclic aromatic hydrocarbons (PAHs). A significant portion (10%-20%) of the Milky Way's carbon reservoir is locked in PAH molecules, which makes their characterization integral to our understanding of astrochemistry. In molecular clouds and the dense envelopes and disks of young stellar objects (YSOs), PAHs are expected to be frozen in the icy mantles of dust grains where they should reveal themselves through infrared absorption. To facilitate the search for frozen interstellar PAHs, laboratory experiments were conducted to determine the positions and strengths of the bands of pyrene mixed with H2O and D2O ices. The D2O mixtures are used to measure pyrene bands that are masked by the strong bands of H2O, leading to the first laboratory determination of the band strength for the CH stretching mode of pyrene in water ice near 3.25 microns. Our infrared band strengths were normalized to experimentally determined ultraviolet band strengths, and we find that they are generally ~50% larger than those reported by Bouwman et al. based on theoretical strengths. These improved band strengths were used to reexamine YSO spectra published by Boogert et al. to estimate the contribution of frozen PAHs to absorption in the 5-8 micron spectral region, taking into account the strength of the 3.25 micron CH stretching mode. It is found that frozen neutral PAHs contain 5%-9% of the cosmic carbon budget, and account for 2%-9% of the unidentified absorption in the 5-8 micron region.