Observations show that 100-km-class Kuiper belt objects (KBOs) can be divided in (at least) two color groups, hereafter red (R, g-i<1.2) and very red (VR, g-i>1.2), reflecting a difference in their surface composition. This is thought to imply that KBOs formed over a relatively wide range of radial distance, r. The cold classicals at 42<r<47 au are predominantly VR and known Neptune Trojans at r=30 au are mostly R. Intriguingly, however, the dynamically hot KBOs show a mix of R and VR colors and no correlation of color with r. Here we perform migration/instability simulations where the Kuiper belt is populated from an extended planetesimal disk. We find that the color observations can be best understood if R objects formed at r<r* and VR objects at r>r*, with 30<r*<40 au. The proposed transition at 30<r*<40 au would explain why the VR objects in the dynamically hot population have smaller orbital inclinations than the R objects, because the orbital excitation from Neptune weakens for orbits starting beyond 30 au. Possible causes of the R-VR color bimodality are discussed.