The major source of noise in high-contrast imaging is the presence of slowly evolving speckles that do not average with time. The temporal stability of the point-spread-function (PSF) is therefore critical to reach a high contrast with extreme adaptive optics (xAO) instruments. Understanding on which timescales the PSF evolves and what are the critical parameters driving the speckle variability allow to design an optimal observing strategy and data reduction technique to calibrate instrumental aberrations and reveal faint astrophysical sources. We have obtained a series of 52 min, AO-corrected, coronagraphically occulted, high-cadence (1.6Hz), H-band images of the star HR 3484 with the SPHERE (Spectro-Polarimeter High-contrast Exoplanet REsearch instrument on the VLT. This is a unique data set from an xAO instrument to study its stability on timescales as short as one second and as long as several tens of minutes. We find different temporal regimes of decorrelation. We show that residuals from the atmospheric turbulence induce a fast, partial decorrelation of the PSF over a few seconds, before a transition to a regime with a linear decorrelation with time, at a rate of several tens parts per million per second (ppm/s). We analyze the spatial dependence of this decorrelation, within the well-corrected radius of the adaptive optics system and show that the linear decorrelation is faster at short separations. Last, we investigate the influence of the distance to the meridian on the decorrelation.