An important aspect of quenching star formation is the removal of the cold interstellar medium (ISM; non-ionised gas and dust) from a galaxy. In addition, dust grains can be destroyed in a hot or turbulent medium. The adopted timescale of dust removal usually relies on uncertain theoretical estimates. It is tricky to track the dust removal, because usually dust is constantly replenished by consecutive generations of stars. Our objective is to measure observationally the timescale of dust removal. We here explore an approach to select galaxies which do have detectable amounts of dust and cold ISM but exhibit a low current dust production rate. Any decrease of the dust and gas content as a function of the age of such galaxies therefore must be attributed to processes governing the ISM removal. We used a sample of galaxies detected by Herschel in the far-infrared with visually assigned early-type morphology or spirals with red colours. We also obtained JCMT/SCUBA-2 observations for five of them. We discovered an exponential decline of the dust-to-stellar mass ratio with age, which we interpret as an evolutionary trend of dust removal from these galaxies. For the first time we directly measure the dust removal timescale in such galaxies to be tau=(2.5+-0.4) Gyr (the corresponding half-life time is (1.75+-0.25) Gyr). This quantity may be used in models in which it must be assumed a priori and cannot be derived. Any process which removes dust in these galaxies, such as dust grain destruction, cannot happen on shorter timescales. The timescale is comparable to the quenching timescales found in simulations for galaxies with similar stellar masses. The dust is likely of internal, not external origin. It was either formed in the past directly by supernovae, or from seeds produced by SNe and with grain growth in the ISM contributing substantially to the dust mass accumulation.