Models of galaxy formation invoke the major merger of gas-rich progenitor galaxies as the trigger for significant phases of black hole growth and the associated feedback that suppresses star formation to create red spheroidal remnants. However, the observational evidence for the connection between mergers and active galactic nucleus (AGN) phases is not clear. We analyze a sample of low-mass early-type galaxies known to be in the process of migrating from the blue cloud to the red sequence via an AGN phase in the green valley. Using deeper imaging from SDSS Stripe 82, we show that the fraction of objects with major morphological disturbances is high during the early starburst phase, but declines rapidly to the background level seen in quiescent early-type galaxies by the time of substantial AGN radiation several hundred Myr after the starburst. This observation empirically links the AGN activity in low-redshift early-type galaxies to a significant merger event in the recent past. The large time delay between the merger-driven starburst and the peak of AGN activity allows for the merger features to decay to the background and hence may explain the weak link between merger features and AGN activity in the literature.