X-ray polarimetry has seen a growing interest in recent years. Improvements in detector technology and focusing X-ray optics now enable sensitive astrophysical X-ray polarization measurements. These measurements will provide new insights into the processes at work in accreting black holes, the emission of X-rays from neutron stars and magnetars, and the structure of AGN jets. X-Calibur is a balloon-borne hard X-ray scattering polarimeter. An X-ray mirror with a focal length of 8 m focuses X-rays onto the detector, which consists of a plastic scattering element surrounded by Cadmium-Zinc-Telluride detectors, which absorb and record the scattered X-rays. Since X-rays preferentially scatter perpendicular to their polarization direction, the polarization properties of an X-ray beam can be inferred from the azimuthal distribution of scattered X-rays. A close alignment of the X-ray focal spot with the center of the detector is required in order to reduce systematic uncertainties and to maintain a high photon detection efficiency. This places stringent requirements on the mechanical and thermal stability of the telescope structure. During the flight on a stratospheric balloon, X-Calibur makes use of the Wallops Arc-Second Pointer (WASP) to point the telescope at astrophysical sources. In this paper, we describe the design, construction, and test of the telescope structure, as well as its performance during a 25-hour flight from Ft. Sumner, New Mexico. The carbon fiber-aluminum composite structure met the requirements set by X-Calibur and its design can easily be adapted for other types of experiments, such as X-ray imaging or spectroscopic telescopes.