The dynamical behavior of spacecraft around asteroids is a key element in design of such missions. An asteroid's irregular shape, non-spherical mass distribution and its rotational sate make the dynamics of spacecraft quite complex. This paper focuses on the gravity gradient torque of spacecraft around non-spherical asteroids. The gravity field of the asteroid is approximated as a 2nd degree and order-gravity field with harmonic coefficients C20 and C22. By introducing the spacecraft's higher-order inertia integrals, a full fourth-order gravity gradient torque model of the spacecraft is established through the gravitational potential derivatives. Our full fourth-order model is more precise than previous fourth-order model due to the consideration of higher-order inertia integrals of the spacecraft. Some interesting conclusions about the gravity gradient torque model are reached. Then a numerical simulation is carried out to verify our model. In the numerical simulation, a special spacecraft consisted of 36 point masses connected by rigid massless rods is considered. We assume that the asteroid is in a uniform rotation around its maximum-moment principal axis, and the spacecraft is on the stationary orbit in the equatorial plane. Simulation results show that the motion of previous fourth-order model is quite different from the exact motion, while our full fourth-order model fits the exact motion very well. And our model is precise enough for practical applications.