We explore the relations between size, stellar mass and average stellar population age (indicated by D$_n4000$ indices) for a sample of $\sim11000$ intermediate-redshift galaxies from the SHELS spectroscopic survey (Geller et al. 2014) augmented by high-resolution Subaru Telescope Hyper Suprime-Cam imaging. In the redshift interval $0.1<z<0.6$, star forming galaxies are on average larger than their quiescent counterparts. The mass-complete sample of $\sim3500$ $M_*>10^{10}\, M_\odot$ quiescent galaxies shows that the average size of a $10^{11}\, M_\odot$ quiescent galaxy increases by $\lesssim25\%$ from $z\sim0.6$ to $z\sim0.1$. This growth rate is a function of stellar mass: the most massive ($M_*>10^{11}\, M_\odot$) galaxies grow significantly more slowly in size than an order of magnitude less massive quiescent systems that grow by 70\% in the $0.1\lesssim z\lesssim0.3$ redshift interval. For $M_*<10^{11}\, M_\odot$ galaxies age and size are anti-correlated at fixed mass; more massive quiescent systems show no significant trend in size with average stellar population age. The evolution in absolute and fractional abundances of quiescent systems at intermediate redshift are also a function of galaxy stellar mass. The suite of evolutionary trends suggests that galaxies more massive than $\sim10^{11}\, M_\odot$ have mostly assembled their mass by $z\sim0.6$. Quiescent galaxies with lower stellar masses show more complex evolution that is characterized by a combination of individual quiescent galaxy size growth (through mergers) and an increase in the size of newly quenched galaxies joining the population at later times (progenitor bias). The $M_*\sim10^{10}\, M_\odot$ population grows predominantly as a result of progenitor bias. For $M_*\sim5\times10^{10}\, M_\odot$ quiescent galaxies, mergers and progenitor bias make more comparable contributions to the size growth.[abridged]