We address the effect of particle scattering on the energy spectra of solar energetic electron events using i) an observational and ii) a modeling approach. i) We statistically study observations of the STEREO spacecraft making use of directional electron measurements made with the SEPT instrument in the range of 45 -- 425 keV. We compare the energy spectra of the anti-sunward propagating beam with that one of the backward scattered population and find that, on average, the backward scattered population shows a harder spectrum with the effect being stronger at higher energies. ii) We use a numerical SEP transport model to simulate the effect of particle scattering (both in terms of pitch-angle and perpendicular to the mean field) on the spectrum. We find that pitch-angle scattering can lead to spectral changes at higher energies (E $>100$ keV) and further away from the Sun (r $> 1$ au) which are also often observed. At lower energies, and closer to the Sun the effect of pitch-angle scattering is much reduced so that the simulated energy spectra still resemble the injected power-law functions. When examining pitch-angle dependent spectra, we find, in agreement with the observational results, that the spectra of the backward propagating electrons are harder than that of the forward (from the Sun) propagating population. {We conclude that {\it Solar Orbiter} and {\it Parker Solar Probe} will be able to observe the unmodulated omni-directional SEP electron spectrum close to the Sun at higher energies, giving a direct indication of the accelerated spectrum. }