Recent hydrodynamical simulations predict that stellar feedback in intermediate mass galaxies (IMGs) can drive strong fluctuations in structure (e.g., half-light radius, $R_e$). This process operates on timescales of only a few hundred Myr and persists even at late cosmic times. One prediction of this quasi-periodic, galactic-scale "breathing" is an $anti$-correlation between star formation rate (SFR) and half-light radius as central gas overdensities lead to starbursts whose feedback drags stars to larger radii while star formation dwindles. We test this prediction with a sample of 322 $isolated$ IMGs with stellar masses of $10^{9.0} \leq M/M_{\odot} \leq 10^{9.5}$ at $0.3<z<0.4$ in the HST $I_{814}$ COSMOS footprint. We find that IMGs with higher specific SFRs (SSFR $>10^{-10}$ yr$^{-1}$) are the most extended with median sizes of $R_e \sim 3-3.4$ kpc and are mostly disk-dominated systems. In contrast, IMGs with lower SSFRs ($<10^{-10}$ yr$^{-1}$) are a factor of $\sim 2-3$ more compact with median sizes of $R_e \sim 0.9-1.6$ kpc and have more significant bulge contributions to their light. These observed trends are opposite the predictions for stellar feedback that operate via the "breathing" process described above. We discuss various paths to reconcile the observations and simulations, all of which likely require a different implementation of stellar feedback in IMGs that drastically changes their predicted formation history.