We present results from the fitting of infrared (IR) spectral energy distributions of 21 active galactic nuclei (AGN) with clumpy torus models. We compiled high spatial resolution ($\sim 0.3$--$0.7$ arcsec) mid-IR $N$-band spectroscopy, $Q$-band imaging and nuclear near- and mid-IR photometry from the literature. Combining these nuclear near- and mid-IR observations, far-IR photometry and clumpy torus models, enables us to put constraints on the torus properties and geometry. We divide the sample into three types according to the broad line region (BLR) properties; type-1s, type-2s with scattered or hidden broad line region (HBLR) previously observed, and type-2s without any published HBLR signature (NHBLR). Comparing the torus model parameters gives us the first quantitative torus geometrical view for each subgroup. We find that NHBLR AGN have smaller torus opening angles and larger covering factors than those of HBLR AGN. This suggests that the chance to observe scattered (polarized) flux from the BLR in NHBLR could be reduced by the dual effects of (a) less scattering medium due to the reduced scattering volume given the small torus opening angle and (b) the increased torus obscuration between the observer and the scattering region. These effects give a reasonable explanation for the lack of observed HBLR in some type-2 AGN.