Gamma-ray burst X-ray flares are believed to mark the late time activity of the central engine. We compute the temporal evolution of the average flare luminosity $< L >$ in the common rest frame energy band of 44 GRBs taken from the large \emph{Swift} 5-years data base. Our work highlights the importance of a proper consideration of the threshold of detection of flares against the contemporaneous continuous X-ray emission. In the time interval $30 \rm{s}<t<1000\,\rm{s}$ we find $< L >\propto t^{-2.7\pm 0.1}$; this implies that the flare isotropic energy scaling is $E_{\rm{iso,flare}}\propto t^{-1.7}$. The decay of the continuum underlying the flare emission closely tracks the average flare luminosity evolution, with a typical flare to steep-decay luminosity ratio which is $L_{\rm{flare}}/L_{\rm{steep}}=4.7$: this suggests that flares and continuum emission are deeply related to one another. We infer on the progenitor properties considering different models. According to the hyper-accreting black hole scenario, the average flare luminosity scaling can be obtained in the case of rapid accretion ($t_{\rm{acc}}\ll t$) or when the last $\sim 0.5 M_{\sun}$ of the original $14 M_{\sun}$ progenitor star are accreted. Alternatively, the steep $\propto t^{-2.7}$ behaviour could be triggered by a rapid outward expansion of an accretion shock in the material feeding a convective disk. If instead we assume the engine to be a rapidly spinning magnetar, then its rotational energy can be extracted to power a jet whose luminosity is likely to be between the monopole ($L\propto e^{-2t}$) and dipole ($L\propto t^{-2}$) cases. In both scenarios we suggest the variability, which is the main signature of the flaring activity, to be established as a consequence of different kinds of instabilities.