We examine the ability of gravitational lens time delays to reveal complex structure in lens potentials. In Congdon, Keeton & Nordgren (2008), we predicted how the time delay between the bright pair of images in a "fold" lens scales with the image separation, for smooth lens potentials. Here we show that the proportionality constant increases with the quadrupole moment of the lens potential, and depends only weakly on the position of the source along the caustic. We use Monte Carlo simulations to determine the range of time delays that can be produced by realistic smooth lens models consisting of isothermal ellipsoid galaxies with tidal shear. We can then identify outliers as "time delay anomalies". We find evidence for anomalies in close image pairs in the cusp lenses RX J1131$-$1231 and B1422+231. The anomalies in RX J1131$-$1231 provide strong evidence for substructure in the lens potential, while at this point the apparent anomalies in B1422+231 mainly indicate that the time delay measurements need to be improved. We also find evidence for time delay anomalies in larger-separation image pairs in the fold lenses, B1608+656 and WFI 2033$-$4723, and the cusp lens RX J0911+0551. We suggest that these anomalies are caused by some combination of substructure and a complex lens environment. Finally, to assist future monitoring campaigns we use our smooth models with shear to predict the time delays for all known four-image lenses.

Gravitational lensing provides a unique and powerful probe of the mass distributions of distant galaxies. Four-image lens systems with fold and cusp configurations have two or three bright images near a critical point. Within the framework of singularity theory, we derive analytic relations that are satisfied for a light source that lies a small but finite distance from the astroid caustic of a four-image lens. Using a perturbative expansion of the image positions, we show that the time delay between the close pair of images in a fold lens scales with the cube of the image separation, with a constant of proportionality that depends on a particular third derivative of the lens potential. We also apply our formalism to cusp lenses, where we develop perturbative expressions for the image positions, magnifications and time delays of the images in a cusp triplet. Some of these results were derived previously for a source asymptotically close to a cusp point, but using a simplified form of the lens equation whose validity may be in doubt for sources that lie at astrophysically relevant distances from the caustic. Along with the work of Keeton et al. (2005), this paper demonstrates that perturbation theory plays an important role in theoretical lensing studies.