Radial velocities of 2768 fundamental mode RR Lyrae stars (RRLs) toward the Southern Galactic bulge are presented, spanning the southern bulge from -8 < l < +8 and -3 < b <-6. Distances derived from the pulsation properties of the RRLs are combined with Gaia proper motions to give constraints on the orbital motions of 1389 RRLs. The majority (~75%) of the bulge RRLs have orbits consistent with these stars being permanently bound to <3.5 kpc from the Galactic Center, similar to the bar. However, unlike the bulge giants, the RRLs exhibit slower rotation and a higher velocity dispersion. The higher velocity dispersion arises almost exclusively from halo interlopers passing through the inner Galaxy. We present 82 stars with space velocities > 500 km/s and find that the majority of these high-velocity stars are halo interlopers; it is unclear if a sub-sample of these stars with similar space velocities have a common origin. Once the 25% of the sample represented by halo interlopers is cleaned, we can clearly discern two populations of bulge RRLs in the inner Galaxy. One population of RRLs is not as tightly bound to the Galaxy (but is still confined to the inner ~3.5 kpc), and is both spatially and kinematically consistent with the barred bulge. The second population is more centrally concentrated and does not trace the bar. One possible interpretation is that this population was born prior to bar formation, as its spatial location, kinematics and pulsation properties suggest, possibly from an accretion event at high redshift.

Radial velocities of fifteen double-mode bulge RR Lyrae (RR01) stars are presented, six of which belong to a compact group of RR01 stars in pulsation space, with the ratio of first-overtone period to fundamental-mode period, P_{fo}/P_{f}~0.74, and P_{f}~0.44. It has been suggested that these pulsationally clumped RR01 stars are a relic of a disrupted dwarf galaxy or stellar cluster, as they also appear to be spatially coherent in a vertical strip across the bulge. However, the radial velocities of the stars presented here, along with proper motions from Gaia DR2, show a large range of radial velocities, proper motions and distances for the bulge RR01 stars in the pulsation clump, much larger than the RR01 stars in the Sagittarius dwarf galaxy (Sgr). Therefore, in contrast to the kinematics of the RRL stars belonging to Sgr, and those in and surrounding the bulge globular cluster NGC~6441, there is no obvious kinematic signature within the pulsationally clumped RR01 stars. If the pulsationally clumped RR01 stars belonged to the same system in the past and were accreted, their accretion in the inner Galaxy was not recent, as the kinematic signature of this group has been lost (i.e., these stars are now well-mixed within the inner Galaxy). We show that the apparent spatial coherence reported for these stars could have been caused by small number statistics. The orbits of the RR01 stars in the inner Galaxy suggest they are confined to the innermost ~4~kpc of the Milky Way.