Galaxy clusters contain a large population of low mass dwarf elliptical galaxies whose exact origin is unclear: their colors, structural properties and kinematics differ substantially from those of dwarf irregulars in the field. We use the Illustris cosmological simulation to study differences in the assembly paths of dwarf galaxies (3e8 < M_*/M_sun < 1e10) according to their environment. We find that cluster dwarfs achieve their maximum total and stellar mass on average ~ 8 and ~ 4.5 Gyr ago (or redshifts z = 1.0 and z = 0.4, respectively), around the time of infall into the clusters. In contrast, field dwarfs not subjected to environmental stripping, reach their maximum mass at redshift z = 0. This different assembly history naturally produces a color bimodality, with blue isolated dwarfs and redder cluster dwarfs exhibiting negligible star-formation today. The cessation of star formation happens over median times 3.5-5 Gyr depending on stellar mass, and shows a large scatter (~ 1-8 Gyr), with the lower values associated with starburst events that occur at infall through the virial radius or pericentric passages. We argue that such starbursts together with the early assembly of cluster dwarfs can provide a natural explanation for the higher specific frequency of globular clusters (GCs) in cluster dwarfs, as found observationally. We present a simple model for the formation and stripping of GCs that supports this interpretation. The origin of dwarf ellipticals in clusters is, therefore, consistent with an environmentally-driven evolution of field dwarf irregulars. However the z = 0 field analogs of cluster dwarf progenitors have today stellar masses a factor ~ 3 larger --a difference arising from the early truncation of star formation in cluster dwarfs.