A promising mechanism to form intermediate-mass black holes (IMBHs) is the runaway merger in dense star clusters, where main-sequence stars collide and form a very massive star (VMS), which then collapses to a black hole. In this paper we study the effects of primordial mass segregation and the importance of the stellar initial mass function (IMF) on the runaway growth of VMSs using a dynamical Monte Carlo code for N-body systems with N as high as 10^6 stars. Our code now includes an explicit treatment of all stellar collisions. We place special emphasis on the possibility of top-heavy IMFs, as observed in some very young massive clusters. We find that both primordial mass segregation and the shape of the IMF affect the rate of core collapse of star clusters and thus the time of the runaway. When we include primordial mass segregation we generally see a decrease in core collapse time (tcc). Moreover, primordial mass segregation increases the average mass in the core, thus reducing the central relaxation time, which also decreases tcc. The final mass of the VMS formed is always close to \sim 10^-3 of the total cluster mass, in agreement with the previous studies and is reminiscent of the observed correlation between the central black hole mass and the bulge mass of the galaxies. As the degree of primordial mass segregation is increased, the mass of the VMS increases at most by a factor of 3. Flatter IMFs generally increase the average mass in the whole cluster, which increases tcc. For the range of IMFs investigated in this paper, this increase in tcc is to some degree balanced by stellar collisions, which accelerate core collapse. Thus there is no significant change in tcc for the somewhat flatter global IMFs observed in very young massive clusters.