Radial velocities of 2768 fundamental mode RR Lyrae stars (RRLs) toward the Southern Galactic bulge are presented, spanning the southern bulge from -8 < l < +8 and -3 < b <-6. Distances derived from the pulsation properties of the RRLs are combined with Gaia proper motions to give constraints on the orbital motions of 1389 RRLs. The majority (~75%) of the bulge RRLs have orbits consistent with these stars being permanently bound to <3.5 kpc from the Galactic Center, similar to the bar. However, unlike the bulge giants, the RRLs exhibit slower rotation and a higher velocity dispersion. The higher velocity dispersion arises almost exclusively from halo interlopers passing through the inner Galaxy. We present 82 stars with space velocities > 500 km/s and find that the majority of these high-velocity stars are halo interlopers; it is unclear if a sub-sample of these stars with similar space velocities have a common origin. Once the 25% of the sample represented by halo interlopers is cleaned, we can clearly discern two populations of bulge RRLs in the inner Galaxy. One population of RRLs is not as tightly bound to the Galaxy (but is still confined to the inner ~3.5 kpc), and is both spatially and kinematically consistent with the barred bulge. The second population is more centrally concentrated and does not trace the bar. One possible interpretation is that this population was born prior to bar formation, as its spatial location, kinematics and pulsation properties suggest, possibly from an accretion event at high redshift.