We present our ALMA Cycle 4 measurements of the [CII] emission line and the underlying far-infrared (FIR) continuum emission from four optically low-luminosity ($M_{\rm 1450} > -25$) quasars at $z \gtrsim 6$ discovered by the Subaru Hyper Suprime Cam (HSC) survey. The [CII] line and FIR continuum luminosities lie in the ranges $L_{\rm [CII]} = (3.8-10.2) \times 10^8~L_\odot$ and $L_{\rm FIR} = (1.2-2.0) \times 10^{11}~L_\odot$, which are at least one order of magnitude smaller than those of optically-luminous quasars at $z \gtrsim 6$. We estimate the star formation rates (SFR) of our targets as $\simeq 23-40~M_\odot ~{\rm yr}^{-1}$. Their line and continuum-emitting regions are marginally resolved, and found to be comparable in size to those of optically luminous quasars, indicating that their SFR or likely gas mass surface densities (key controlling parameter of mass accretion) are accordingly different. The $L_{\rm [CII]}/L_{\rm FIR}$ ratios of the hosts, $\simeq (2.2-8.7) \times 10^{-3}$, are fully consistent with local star-forming galaxies. Using the [CII] dynamics, we derived their dynamical masses within a radius of 1.5-2.5 kpc as $\simeq (1.4-8.2) \times 10^{10}~M_\odot$. By interpreting these masses as stellar ones, we suggest that these faint quasar hosts are on or even below the star-forming main sequence at $z \sim 6$, i.e., they appear to be transforming into quiescent galaxies. This is in contrast to the optically luminous quasars at those redshifts, which show starburst-like properties. Finally, we find that the ratios of black hole mass to host galaxy dynamical mass of the most of low-luminosity quasars including the HSC ones are consistent with the local value. The mass ratios of the HSC quasars can be reproduced by a semi-analytical model that assumes merger-induced black hole-host galaxy evolution.