Observations indicate blue globular clusters have half-light radii systematically larger than those of red globular clusters. In this paper, we test whether the different metallicity-dependent stellar evolution timescales and mass-loss rates within the clusters can impact their early dynamical evolution. By means of N-body simulations including stellar evolution recipes we simulate the early evolution of small centrally concentrated clusters with and without primordial mass segregation. Our simulations include accurate metallicity-dependent mass loss from massive stars. We find blue clusters to be larger than red clusters regardless of whether the clusters have been primordially mass segregated. In addition, the size difference is found to be larger and consistent with observations for initial models with a low central concentration. These results indicate that the systematic size difference found between red and blue clusters can, at least in part, be attributed to the dynamical effects of differing stellar evolution histories, driven by metallicity.