The recently observed X-ray synchrotron emission from four supernova remnants (SNR) has strengthened the evidence that cosmic ray electrons are accelerated in SNR. We show, that if this is indeed the case, the local electron spectrum will be strongly time-dependent, at least above roughly 30 GeV. The time dependence stems from the Poisson fluctuations in the number of SNR within a certain volume and within a certain time interval. As far as cosmic ray electrons are concerned, the Galaxy looks like actively bubbling swiss cheese rather than a steady, homogeneously filled system. Our finding has important consequences for studies of the Galactic diffuse gamma-ray emission, for which a strong excess over model predictions above 1 GeV has been reported recently. While these models were relying on an electron injection spectrum with index 2.4 -- chosen to fit the local electron flux up to 1 TeV -- we show that an electron injection index of around 2.0 would a) be consistent with the expected Poisson fluctuations in the locally observable electron spectrum and b) explain the above mentioned gamma-ray excess above 1 GeV. An electron injection index around 2 would also correspond to the average radio synchrotron spectrum of individual SNR. We use a three-dimensional propagation code to calculate the spectra of electrons throughout the Galaxy and show that the longitude and latitude distribution of the leptonic gamma-ray production above 1 GeV is in accord with the respective distributions for the gamma-ray excess. We finally point out that our model implies a strong systematic uncertainty in the determination of the spectrum of the extragalactic gamma-ray background.