We present new constraints on the couplings of axions and more generic axion-like particles using data from the EDELWEISS-II experiment. The EDELWEISS experiment, located at the Underground Laboratory of Modane, primarily aims at the direct detection of WIMPs using germanium bolometers. It is also sensitive to the low-energy electron recoils that would be induced by solar or dark matter axions. Using a total exposure of up to 448 kg.d, we searched for axion-induced electron recoils down to 2.5 keV within four scenarios involving different hypotheses on the origin and couplings of axions. We set a 95% CL limit on the coupling to photons $g_{A\gamma}<2.13\times 10^{-9}$ GeV$^{-1}$ in a mass range not fully covered by axion helioscopes. We also constrain the coupling to electrons, $g_{Ae} < 2.56\times 10^{-11}$, similar to the more indirect solar neutrino bound. Finally we place a limit on $g_{Ae}\times g_{AN}^{\rm eff}<4.70 \times 10^{-17}$, where $g_{AN}^{\rm eff}$ is the effective axion-nucleon coupling for $^{57}$Fe. Combining these results we fully exclude the mass range $0.91\,{\rm eV}<m_A<80$ keV for DFSZ axions and $5.73\,{\rm eV}<m_A<40$ keV for KSVZ axions.