Ever since the Apollo era, a question has remained as to the origin of the lunar swirls (high albedo regions coincident with the regions of surface magnetization). Different processes have been proposed for their origin. In this work we test the idea that the lunar swirls have a higher albedo relative to surrounding regions because they deflect incoming solar wind particles that would otherwise darken the surface. 3D particle tracking is used to estimate the influence of five lunar magnetic anomalies on incoming solar wind. The regions investigated include Mare Ingenii, Gerasimovich, Renier Gamma, Northwest of Apollo and Marginis. Both protons and electrons are tracked as they interact with the anomalous magnetic field and impact maps are calculated. The impact maps are then compared to optical observations and comparisons are made between the maxima and minima in surface fluxes and the albedo and optical maturity of the regions. Results show deflection of slow to typical solar wind particles on a larger scale than the fine scale optical, swirl, features. It is found that efficiency of a particular anomaly for deflection of incoming particles does not only scale directly with surface magnetic field strength, but also is a function of the coherence of the magnetic field. All anomalous regions can also produce moderate deflection of fast solar wind particles. The anomalies' influence on ~ 1 GeV SEP particles is only apparent as a slight modification of the incident velocities.