We present a detailed millimeter line study of the circumstellar environment of the low-luminosity Class 0 protostar IRAM 04191+1522 in the Taurus molecular cloud. New line observations demonstrate that the ~14000 AU radius protostellar envelope is undergoing both extended infall and fast, differential rotation. Radiative transfer modeling of multitransition CS and C34S maps indicate an infall velocity v_inf ~ 0.15 km/s at r ~ 1500 AU and v_inf ~ 0.1 km/s up to r ~ 11000 AU, as well as a rotational angular velocity Omega ~ 3.9 x 10^{-13} rad/s, strongly decreasing with radius beyond 3500 AU down to a value Omega ~ 1.5-3 x 10^{-14} rad/s at ~ 11000 AU. Two distinct regions, which differ in both their infall and their rotation properties, therefore seem to stand out: the inner part of the envelope (r ~< 2000-4000 AU) is rapidly collapsing and rotating, while the outer part undergoes only moderate infall/contraction and slower rotation. These contrasted features suggest that angular momentum is conserved in the collapsing inner region but efficiently dissipated due to magnetic braking in the slowly contracting outer region. We propose that the inner envelope is in the process of decoupling from the ambient cloud and corresponds to the effective mass reservoir (~0.5 M_sun) from which the central star is being built. Comparison with the rotational properties of other objects in Taurus suggests that IRAM 04191 is at a pivotal stage between a prestellar regime of constant angular velocity enforced by magnetic braking and a dynamical, protostellar regime of nearly conserved angular momentum. The rotation velocity profile we derive for the inner IRAM 04191 envelope should thus set some constraints on the distribution of angular momentum on the scale of the outer Solar system at the onset of protostar/disk formation.