The Milky Way can act as a large-scale weak gravitational lens of the cosmic microwave background (CMB). We study this effect using a photon ray-tracing code and a Galactic mass distribution with disk, bulge and halo components. For an observer at the Sun's coordinates in the Galaxy, the bending of CMB photon paths is limited to less than one arcsecond, and only for rays that pass within a few degrees of the Galactic Center. However, the entire sky is affected, resulting in global distortions of the CMB on large angular scales. These distortions can cause the low-order multipoles of a spherical harmonic expansion of the CMB sky temperature to leak into higher-order modes. Thus the component of the CMB dipole that results from the Local Group's motion relative to the local cosmic frame of rest contributes to higher-order moments for an observer in the solar system. With our ray-tracing code we show that the phenomenon is not sensitive to the specific choice of Galactic potential. We also quantitatively rule it out as a contributor to CMB anomalies such as power asymmetry or correlated alignment of low-order multipole moments.