We use the Spitzer Wide-area InfraRed Extragalactic Legacy Survey (SWIRE) to explore the specific star-formation activity of galaxies and their evolution near the peak of the cosmic far-infrared (FIR) background at 70 and 160um. We use a stacking analysis to determine the mean FIR properties of well defined subsets of galaxies at flux levels well below the FIR catalogue detection limits of SWIRE and other Spitzer surveys. We tabulate the contribution of different subsets of galaxies to the FIR background at 70um and 160um. These long wavelengths provide a good constraint on the bolometric, obscured emission. The large area provides good constraints at low z and in finer redshift bins than previous work. At all redshifts we find that the specific FIR Luminosity (sLFIR) decreases with increasing mass, following a trend L_FIR/M* propto M_* ^beta with beta =-0.38\pm0.14. This is a more continuous change than expected from the {Delucia2007} semi-analytic model suggesting modifications to the feedback prescriptions. We see an increase in the sLFIR by about a factor of ~100 from 0<z<2 and find that the sLFIR evolves as (1+z)^alpha with alpha=4.4\pm0.3 for galaxies with 10.5 < log M*/Msun < 12. This is considerably steeper than the {Delucia2007} semi-analytic model (alpha \sim 2.5). When separating galaxies into early and late types on the basis of the optical/IR spectral energy distributions we find that the decrease in sLFIR with stellar mass is stronger in early type galaxies (beta ~ -0.46), while late type galaxies exhibit a flatter trend (beta \sim -0.15). The evolution is strong for both classes but stronger for the early type galaxies. The early types show a trend of decreasing strength of evolution as we move from lower to higher masses while the evolution of the late type galaxies has little dependence on stellar mass. We suggest that in late-type galaxies we are seeing a consistently declining sSFR..