The STEREO/WAVES (SWAVES) experiment on board the two STEREO spacecraft (Solar Terrestrial Relations Observatory) launched on 25 October 2006 is dedicated to the measurement of the radio spectrum at frequencies between a few kilohertz and 16 MHz. The SWAVES antenna system consists of 6 m long orthogonal monopoles designed to measure the electric component of the radio waves. With this configuration direction finding of radio sources and polarimetry (analysis of the polarization state) of incident radio waves is possible. For the evaluation of the SWAVES data the receiving properties of the antennas, distorted by the radiation coupling with the spacecraft body and other onboard devices, have to be known accurately. In the present context, these properties are described by the antenna effective length vectors. We present the results of an in-flight calibration of the SWAVES antennas using the observations of the nonthermal terrestrial auroral kilometric radiation (AKR) during STEREO roll maneuvers in an early stage of the mission. A least squares method combined with a genetic algorithm was applied to find the effective length vectors of the STEREO Behind (STEREO-B)/WAVES antennas in a quasi-static frequency range ($L_{antenna} \ll \lambda_{wave}$) which fit best to the model and observed AKR intensity profiles. The obtained results confirm the former SWAVES antenna analysis by rheometry and numerical simulations. A final set of antenna parameters is recommended as a basis for evaluations of the SWAVES data.