Recent observations of distant disk galaxies show that there is little to no evolution in the relation between maximum rotation speed and stellar mass at z < 1.2. There is however a significant scatter between these two quantities whose origin is uncertain. We show in this paper that this scatter is at least partially the result of galaxy merging, revealing that disk galaxy growth at z < 1 is fundamentally hierarchical. We carry this out by calculating CAS (concentration, asymmetry, clumpiness) structural parameters using archival Hubble Space Telescope imaging of 91 high-redshift disk galaxies at 0.4 < z < 1.0 with robustly measured stellar masses and rotational maximum velocities taken from Conselice et al. (2005). We separate our sample into two redshift bins divided at z = 0.7, and investigate deviations from the stellar-mass Tully-Fisher relation in both the M_* and V_max directions, and how these correlate with structural asymmetries. We find a significant (> 3 sigma) correlation between the residuals from the stellar-mass Tully-Fisher relation in both the M_* and V_max directions, and high asymmetries. This result holds after we remove contributions from star formation and edge-on galaxies which can produce higher asymmetries unrelated to merging. While there are a few cases in which our disk galaxies have very large asymmetries, and are potentially involved in major mergers, in general these asymmetries are smaller than the major merger limit. It is therefore likely that these galaxies are forming hierarchically through minor galaxy mergers, which is also suggested by the constant slope and zero point of the stellar mass Tully-Fisher relation during the same epoch.