We have obtained the first broad-band X-ray spectra of the nearby compact elliptical galaxy M32 by using the ASCA satellite. The extracted spectra and X-ray luminosity are consistent with the properties of the hard spectral component measured in giant elliptical galaxies believed to originate from X-ray binaries. Two ASCA observations were performed two weeks apart; a 25% flux decrease and spectral softening occurred in the interval. We have also analyzed archival ROSAT HRI data, and discovered that the X-ray emission is dominated by a single unresolved source offset from the nucleus of M32. We argue that this offset, combined with the extremely rapid large magnitude variations, and hard X-ray spectrum combine to weakly favor a (single) X-ray binary over an AGN origin for the X-rays from M32. The nuclear black hole in M32 must be fuel-starved and/or accreting from a radiatively inefficient advection-dominated disk: the product of the accretion rate and the radiative efficiency must be less than 1e-10 solar masses per year if the X-ray source is indeed an X-ray binary.