Inflation is most often described using quantum field theory (QFT) on a fixed, curved spacetime background. Such a description is valid only if the spatial volume of the region considered is so large that its size and shape moduli behave classically. However, if we trace an inflating universe back to early times, the volume of any comoving region of interest -- for example the present Hubble volume -- becomes exponentially small. Hence, quantum fluctuations in the trajectory of the background cannot be neglected at early times. In this paper, we develop a path integral description of a flat, inflating patch (approximated as de Sitter spacetime), treating both the background scale factor and the gravitational wave perturbations quantum mechanically. We find this description fails at small values of the initial scale factor, because \emph{two} background saddle point solutions contribute to the path integral. This leads to a breakdown of QFT in curved spacetime, causing the fluctuations to be unstable and out of control. We show the problem may be alleviated by a careful choice of quantum initial conditions, for the background and the fluctuations, provided that the volume of the initial, inflating patch is larger than $\gg H^{-1}$ in Planck units with $H$ the Hubble constant at the start of inflation. The price of the remedy is high: not only the inflating background, but also the stable, Bunch-Davies fluctuations must be input by hand. Our discussion emphasizes that, even if the inflationary scale is far below the Planck mass, new physics is required to explain the initial quantum state of the universe.