Dwarf galaxies play an important role in our understanding of galaxy formation and evolution, and starbursts are believed to strongly affect their structure and evolution. Here we present a study of 12 of the nearest dwarf galaxies thought to be undergoing starbursts as selected primarily by morphology. We show that these "blue amorphous galaxies" are not physically distinguishable from dwarfs selected as starbursting by other methods, such as blue compact dwarfs and HII galaxies. All these classes exhibit exponential surface brightness profiles in their outer regions but often have a central blue excess. Typically, these starbursting "cores" are young (0.01 - 0.1 Gyr) events compared to the older (1 - 10 Gyr) enveloping galaxy. The ratio of the core-to-envelope blue fluxes ranges from essentially zero to about two. These starbursts are therefore modest events involving only a few percent of the stellar mass. The envelopes have surface-brightnesses that are much higher than typical dwarf Irregular (dI) galaxies, so it is unlikely that there is a straightforward evolutionary relation between typical dIs and dwarf starburst galaxies. Instead BAGs may repeatedly cycle through starburst and quiescent phases, corresponding to the galaxies with strong and weak/absent cores respectively. Once BAGs use up the available gas (either through star-formation or galactic winds) so that star-formation is shut off, the faded remnants would strongly resemble dwarf elliptical (dE) galaxies. However, in the current cosmological epoch this is a slow process that is the aftermath of a series of many weak, recurring bursts. Present-day dEs must have experienced more rapid and intense evolution than this in the distant past. (Abridged)