We present measurements of the auto- and cross-frequency power spectra of the cosmic infrared background (CIB) at 250, 350, and 500um (1200, 860, and 600 GHz) from observations totaling ~ 70 deg^2 made with the SPIRE instrument aboard the Herschel Space Observatory. We measure a fractional anisotropy dI / I = 14 +- 4%, detecting signatures arising from the clustering of dusty star-forming galaxies in both the linear (2-halo) and non-linear (1-halo) regimes; and that the transition from the 2- to 1-halo terms, below which power originates predominantly from multiple galaxies within dark matter halos, occurs at k_theta ~ 0.1 - 0.12 arcmin^-1 (l ~ 2160 - 2380), from 250 to 500um. New to this paper is clear evidence of a dependence of the Poisson and 1-halo power on the flux-cut level of masked sources --- suggesting that some fraction of the more luminous sources occupy more massive halos as satellites, or are possibly close pairs. We measure the cross-correlation power spectra between bands, finding that bands which are farthest apart are the least correlated, as well as hints of a reduction in the correlation between bands when resolved sources are more aggressively masked. In the second part of the paper we attempt to interpret the measurements in the framework of the halo model. With the aim of fitting simultaneously with one model the power spectra, number counts, and absolute CIB level in all bands, we find that this is achievable by invoking a luminosity-mass relationship, such that the luminosity-to-mass ratio peaks at a particular halo mass scale and declines towards lower and higher mass halos. Our best-fit model finds that the halo mass which is most efficient at hosting star formation in the redshift range of peak star-forming activity, z ~ 1-3, is log(M_peak/M_sun) ~ 12.1 +- 0.5, and that the minimum halo mass to host infrared galaxies is log(M_min/M_sun) ~ 10.1 +- 0.6.