We present the first measurement of clustering properties of low mass galaxies with a stellar mass down to M_*~10^9 Msun at 1<z<4 in 24.4 arcmin^2 of the GOODS-North region with a depth of K_{AB}~25, based on the near infrared observations performed with MOIRCS at the Subaru Telescope. The correlation amplitude strongly depends on the K-band flux, color, and stellar mass of the galaxies. We find that K-band luminous galaxies have a larger correlation length than K-band faint galaxies. For color selected samples at 2<z<4, distant red galaxies with J-K>1.3 show a large bias of b~7.2+-1.3 on scales up to \theta~100" or 3.1 comoving Mpc, while blue galaxies with 0.5<J-K<1.3, in which most Lyman break galaxies are populated, have a weak clustering signal on large scales, but a possible strong small scale excess at \theta<10". For massive galaxies with M_*>~10^{10} Msun, we estimate a correlation length and bias to be r_0~4.5 h^{-1} Mpc and b=1.9-3.5, which are much larger than those of low mass (M_*~10^9-10^{10} Msun) galaxies. The comparison of our measurements with analytic CDM models constrains the properties of hosting dark halos, and indicates that the low mass galaxies would be progenitors of galaxies with a typical luminosity of L<~L_* in the local Universe. The blue galaxies in low mass samples are more strongly clustered in more massive halos with higher occupation numbers than low mass red galaxies. This fact suggests an environment effect due to the halo mass on star formation activity at high-z.