We study matter-wave interferometry in the presence of a stochastic background of gravitational waves. It is shown that if the background has a scale-invariant spectrum over a wide bandwidth (which is expected in a class of inflationary models of Big Bang cosmology), then separated-path interference cannot be observed for a lump of matter of size above a limit which is very insensitive to the strength and bandwidth of the fluctuations, unless the interferometer is servo-controlled or otherwise protected. For ordinary solid matter this limit is of order 1--10 mm. A servo-controlled or cross-correlated device would also exhibit limits to the observation of macroscopic interference, which we estimate for ordinary matter moving at speeds small compared to c.