In recent years proper motion measurements have been added to the set of observational constraints on the current properties of Galactic X-ray binaries. We develop an analysis that allows us to consider all this available information and reconstruct the full evolutionary history of X-ray binaries back to the time of compact object formation. This analysis accounts for mass transfer through the ongoing X-ray phase, tidal circularization before the onset of Roche-lobe overflow, motion through the Galactic potential after the formation of the compact object, and binary orbital dynamics and hydrodynamic modeling of the core collapse. We apply the analysis to the soft X-ray transient GRO J1655-40 and, for the first time, use its full 3D peculiar velocity constraints right after core collapse instead of lower limits on the current space velocity given by the present-day radial velocity. We find that the system has remained within 200pc from the Galactic plane throughout its entire life time and that the mass loss and a kick possibly associated with the black hole (BH) formation imparted a kick velocity of 45-115 km/s to the binary's center of mass. Right after BH formation, the system consists of a 3.5-6.3 Msun BH and a 2.3-4 Msun main-sequence star. At the onset of the X-ray phase the donor is still on the main sequence. We find that a symmetric BH formation event cannot be formally excluded, but that the associated system parameters are only marginally consistent with the currently observed binary properties. BH formation mechanisms involving an asymmetric supernova explosion with associated BH kick velocities of a few tens of km/s, on the other hand, satisfy the constraints much more comfortably. We also derive an upper limit on the BH kick magnitude of 210 km/s. (abridged)