We present the relation between galaxy structure and spectral type, using a K-selected galaxy sample at 0.5<z<2.0. Based on similarities between the UV-to-NIR spectral energy distributions, we classify galaxies into 32 spectral types. The different types span a wide range in evolutionary phases, and thus -- in combination with available CANDELS/F160W imaging -- are ideal to study the structural evolution of galaxies. Effective radii (R_e) and Sersic parameters (n) have been measured for 572 individual galaxies, and for each type, we determine R_e at fixed stellar mass by correcting for the mass-size relation. We use the rest-frame U-V vs. V-J diagram to investigate evolutionary trends. When moving into the direction perpendicular to the star-forming sequence, in which we see the Halpha equivalent width and the specific star formation rate (sSFR) decrease, we find a decrease in R_e and an increase in n. On the quiescent sequence we find an opposite trend, with older redder galaxies being larger. When splitting the sample into redshift bins, we find that young post-starburst galaxies are most prevalent at z>1.5 and significantly smaller than all other galaxy types at the same redshift. This result suggests that the suppression of star formation may be associated with significant structural evolution at z>1.5. At z<1, galaxy types with intermediate sSFRs (10^{-11.5}-10^{-10.5} yr^-1) do not have post-starburst SED shapes. These galaxies have similar sizes as older quiescent galaxies, implying that they can passively evolve onto the quiescent sequence, without increasing the average size of the quiescent galaxy population.