We present the source separation framework SCARLET for multi-band images, which is based on a generalization of the Non-negative Matrix Factorization to alternative and several simultaneous constraints. Our approach describes the observed scene as a mixture of components with compact spatial support and uniform spectra over their support. We present the algorithm to perform the matrix factorization and introduce constraints that are useful for optical images of stars and distinct stellar populations in galaxies, in particular symmetry and monotonicity with respect to the source peak position. We also derive the treatment of correlated noise and convolutions with band-dependent point spread functions, rendering our approach applicable to coadded images observed under variable seeing conditions. SCARLET thus yields a PSF-matched photometry measurement with an optimally chosen weight function given by the mean morphology in all available bands. We demonstrate the performance of SCARLET for deblending crowded extragalactic scenes and on an AGN jet -- host galaxy separation problem in deep 5-band imaging from the Hyper Suprime-Cam Stategic Survey Program. Using simulations with prominent crowding we show that SCARLET yields superior results to the HSC-SDSS deblender for the recovery of total fluxes, colors, and morphologies. Due to its non-parametric nature, a conceptual limitation of SCARLET is its sensitivity to undetected sources or multiple stellar population within detected sources, but an iterative strategy that adds components at the location of significant residuals appears promising. The code is implemented in Python with C++ extensions and is available at https://github.com/fred3m/scarlet