The fluctuation of the angular positions of reference extragalactic radio and optical sources under the influence of the irregular gravitational field of visible Galactic stars is considered. It is shown that these angular fluctuations range from a few up to hundreds of microarcseconds. This leads to a small rotation of the celestial reference frame. The nondiagonal coefficients of the rotation matrix are of the order of a microarcsecond. The temporal variation of these coefficients due to the proper motion of the foreground stars is of the order of one microsecond per 20 years. Therefore, the celestial reference frame can be considered inertial and homogeneous only to microarcsecond accuracy. Astrometric catalogues with microarcsecond accuracy will be unstable, and must be reestablished every 20 years.