The TRAPPIST-1 planetary system represents an exceptional opportunity for the atmospheric characterization of temperate terrestrial exoplanets with the upcoming James Webb Space Telescope (JWST). Assessing the potential impact of stellar contamination on the planets' transit transmission spectra is an essential precursor step to this characterization. Planetary transits themselves can be used to scan the stellar photosphere and to constrain its heterogeneity through transit depth variations in time and wavelength. In this context, we present our analysis of 169 transits observed in the optical from space with K2 and from the ground with the SPECULOOS and Liverpool telescopes. Combining our measured transit depths with literature results gathered in the mid/near-IR with Spitzer/IRAC and HST/WFC3, we construct the broadband transmission spectra of the TRAPPIST-1 planets over the 0.8-4.5 $\mu$m spectral range. While planets b, d, and f spectra show some structures at the 200-300ppm level, the four others are globally flat. Even if we cannot discard their instrumental origins, two scenarios seem to be favored by the data: a stellar photosphere dominated by a few high-latitude giant (cold) spots, or, alternatively, by a few small and hot (3500-4000K) faculae. In both cases, the stellar contamination of the transit transmission spectra is expected to be less dramatic than predicted in recent papers. Nevertheless, based on our results, stellar contamination can still be of comparable or greater order than planetary atmospheric signals at certain wavelengths. Understanding and correcting the effects of stellar heterogeneity therefore appears essential to prepare the exploration of TRAPPIST-1's with JWST.