The final step of the water formation network on interstellar grain surfaces starting from the H + O$_2$ route is the reaction between H and H$_2$O$_2$. This reaction is known to have a high activation energy and therefore at low temperatures it can only proceed via tunneling. To date, however, no rate constants are available at temperatures below 200 K. In this work, we use instanton theory to compute rate constants for the title reaction with and without isotopic substitutions down to temperatures of 50 K. The calculations are based on density functional theory, with additional benchmarks for the activation energy using unrestricted single-reference and multireference coupled-cluster single-point energies. Gas-phase bimolecular rate constants are calculated and compared with available experimental data not only for H + H$_2$O$_2$ $\rightarrow$ H$_2$O + OH, but also for H + H$_2$O$_2$ $\rightarrow$ H$_2$ + HO$_2$. We find a branching ratio where the title reaction is favored by at least two orders of magnitude at 114 K. In the interstellar medium this reaction predominantly occurs on water surfaces, which increases the probability that the two reactants meet. To mimic this one, two, or three spectator H2O molecules are added to the system. Eley-Rideal bimolecular and Langmuir-Hinshelwood unimolecular rate constants are presented here. The kinetic isotope effects for the various cases are compared to experimental data as well as to expressions commonly used in astrochemical models. Both the rectangular barrier and the Eckart approximations lead to errors of about an order of magnitude. Finally, fits of the rate constants are provided as input for astrochemical models.