Many astrophysical and galaxy-scale cosmological problems require a well determined gravitational potential which is often modeled by observers under strong assumptions. Globular clusters (GCs) surrounding galaxies can be used as dynamical tracers of the luminous and dark matter distribution at large (kpc) scales. A natural assumption for modeling the gravitational potential is that GCs accreted in the same dwarf galaxy merger event move at the present time on similar orbits in the host galaxy and should therefore have similar actions. We investigate this idea in one realistic Milky Way like galaxy of the cosmological N-body simulation suite Auriga. We show how the actions of accreted stellar particles in the simulation evolve and that minimizing the standard deviation of GCs in action space, however, cannot constrain the true potential. This approach known as adaptive dynamics does therefore not work for accreted GCs.