SAX J1808.4-3658 is a unique source being the first Low Mass X-ray Binary showing coherent pulsations at a spin period comparable to that of millisecond radio pulsars. Here we present an XMM-Newton observation of SAX J1808.4-3658 in quiescence, the first which assessed its quiescent luminosity and spectrum with good signal to noise. XMM-Newton did not reveal other sources in the vicinity of SAX J1808.4-3658 likely indicating that the source was also detected by previous BeppoSAX and ASCA observations, even if with large positional and flux uncertainties. We derive a 0.5-10 keV unabsorbed luminosity of L_X=5x10^{31} erg/s, a relatively low value compared with other neutron star soft X-ray transient sources. At variance with other soft X-ray transients, the quiescent spectrum of SAX J1808.4-3658 was dominated by a hard (Gamma~1.5) power law with only a minor contribution (<10%) from a soft black body component. If the power law originates in the shock between the wind of a turned-on radio pulsar and matter outflowing from the companion, then a spin-down to X-ray luminosity conversion efficiency of eta~10^{-3} is derived; this is in line with the value estimated from the eclipsing radio pulsar PSR J1740-5340. Within the deep crustal heating model, the faintness of the blackbody-like component indicates that SAX J1808.4-3658 likely hosts a massive neutronstar (M>1.7 solar masses).