We have obtained optical and infrared photometry of the quiescent soft X-ray transient XTE J1118+480. In addition to optical and J-band variations, we present the first observed H- and K_s-band ellipsoidal variations for this system. We model the variations in all bands simultaneously with the WD98 light curve modeling code. The infrared colors of the secondary star in this system are consistent with a K7V, while there is evidence for light from the accretion disk in the optical. Combining the models with the observed spectral energy distribution of the system, the most likely value for the orbital inclination angle is 68 +/- 2 deg. This inclination angle corresponds to a primary black hole mass of 8.53 +/- 0.60 M_sun. Based on the derived physical parameters and infrared colors of the system, we determine a distance of 1.72 +/- 0.10 kpc to XTE J1118+480.