Teleparallel gravity is a formulation of general relativity that is physically equivalent to metric gravity if the gravitational action has the Einstein-Hilbert form and matter is minimally coupled. However, scalar fields generally couple directly to the connection, breaking the equivalence. In particular, this happens for the Standard Model Higgs. We show that a teleparallel theory with a non-minimally coupled scalar field has no linear scalar perturbations, and therefore cannot give successful inflation, unless the non-minimal coupling functions satisfy a particular relation. If the relation is satisfied, Higgs inflation can give an arbitrarily large tensor-to-scalar ratio $r$. Our results also apply to $f(T)$ theories, as they are scalar-tensor theories written in different field coordinates. We discuss generalisation to more complicated actions.