We present observational constraints on the stellar populations of two ultra-diffuse galaxies (UDGs) using optical through near-infrared (NIR) spectral energy distribution (SED) fitting. Our analysis is enabled by new $Spitzer$-IRAC 3.6 $\mu$m and 4.5 $\mu$m imaging, archival optical imaging, and the prospector fully Bayesian SED fitting framework. Our sample contains one field UDG (DGSAT I), one Virgo cluster UDG (VCC 1287), and one Virgo cluster dwarf elliptical for comparison (VCC 1122). We find that the optical--NIR colors of the three galaxies are significantly different from each other. We infer that VCC 1287 has an old ($\gtrsim7.7$ Gyr) and surprisingly metal-poor ($[Z/Z_{\odot}]\lesssim-1.0$) stellar population, even after marginalizing over uncertainties on diffuse interstellar dust. In contrast, the field UDG DGSAT I shows evidence of being younger than the Virgo UDG, with an extended star formation history and an age posterior extending down to $\sim3$ Gyr. The stellar metallicity of DGSAT I is sub-solar but higher than that of the Virgo UDG, with $[Z/Z_{\odot}]=-0.63^{+0.35}_{-0.62}$; in the case of exactly zero diffuse interstellar dust, DGSAT I may even have solar metallicity. With VCC 1287 and several Coma UDGs, a general picture is emerging where cluster UDGs may be "failed" galaxies, but the field UDG DGSAT I seems more consistent with a stellar feedback-induced expansion scenario. In the future, our approach can be applied to a large and diverse sample of UDGs down to faint surface brightness limits, with the goal of constraining their stellar ages, stellar metallicities, and circumstellar and diffuse interstellar dust content.