Spectral transfer processes in magnetohydrodynamic (MHD) turbulence are investigated analytically by decomposition of the velocity and magnetic fields in Fourier space into helical modes. Steady solutions of the dynamical system which governs the evolution of the helical modes are determined, and a stability analysis of these solutions is carried out. The interpretation of the analysis is that unstable solutions lead to energy transfer between the interacting modes while stable solutions do not. From this, a dependence of possible interscale energy and helicity transfers on the helicities of the interacting modes is derived. As expected from the inverse cascade of magnetic helicity in 3D MHD turbulence, mode interactions with like helicities lead to transfer of energy and magnetic helicity to smaller wavenumbers. However, some interactions of modes with unlike helicities also contribute to an inverse energy transfer. As such, an inverse energy cascade for nonhelical magnetic fields is shown to be possible. Furthermore, it is found that high values of the cross-helicity may have an asymmetric effect on forward and reverse transfer of energy, where forward transfer is more quenched in regions of high cross-helicity than reverse transfer. This conforms with recent observations of solar wind turbulence. For specific helical interactions the relation to dynamo action is established.