We analyze and model total solar irradiance variability on time scales from minutes to months, excluding variations due to p-mode oscillations, using a combination of convective and magnetic components. These include granulation, the magnetic network, faculae and sunspots. Analysis of VIRGO data shows that on periods of a day or longer solar variability depends on magnetic activity, but is nearly independent at shorter periods. We assume that only granulation affects the solar irradiance variability on time scales from minutes to hours. Granulation is described as a large sample of bright cells and dark lanes that evolve according to rules deduced from observations and radiation hydrodynamic simulations. Comparison of this model combined with a high time resolution magnetic-field based irradiance reconstruction, with solar data reveals a good correspondence except at periods of 10 to 30 hours. This suggests that the model is missing some power at these periods, which may be due to the absence of supergranulation or insufficient sensitivity of MDI magnetograms used for the reconstruction of the magnetic field-based irradiance reconstructions. Our model also shows that even for spatially unresolved data (such as those available for stars) the Fourier or wavelet transform of time series sampled at high cadence may allow properties of stellar granulation, in particular granule lifetimes to be determined.