We analyze the effects of including $\Delta(1232)$ isobars in an equation of state (EoS) for cold, $\beta$-stable neutron star matter, employing relativistic nuclear mean field theory. The selected EoS reproduces the properties of nuclear matter and finite nuclei and, in the astrophysical context, allows for the presence of hyperons in neutron stars having masses larger than 2$M_{\odot}$. We find that the composition and structure of neutron stars is critically influenced by the addition of the $\Delta$ isobars, which allows us to constrain their interaction with the meson fields taking into account astrophysical information. Imposing that the EoS is stable and ensures the existence of 2$M_{\odot}$ neutron stars, as well as requiring agreement with data of $\Delta$ excitation in nuclei, we find that, in the absence of other mechanisms stiffening the EoS at high densities, the interaction of the $\Delta$ isobars with the sigma and omega meson fields must be at least 10\% stronger than that of the nucleons. Moreover, the neutron star moment of inertia turns out to be sensitive to the presence of $\Delta$ isobars, whereas the inclusion of $\Delta$ isobars in the EoS allows for smaller stellar radii and for a lower value of the tidal deformability consistent with the analysis of the GW170817 merger event.