SiO$_2$ may have been expelled from the core directly following core formation in the early stages of Earth's accretion and onwards through the present day. On account of SiO$_2$'s low density with respect to both the core and the lowermost mantle, we examine the process of SiO$_2$ accumulation at the core-mantle boundary (CMB) and its incorporation into the mantle by buoyant rise. Today, if SiO$_2$ is 100-10000 times more viscous than lower mantle material, the dimensions of SiO$_2$ diapirs formed by the viscous Rayleigh-Taylor instability at the CMB would cause them to be swept into the mantle as inclusions of 100 m - 10 km diameter. Under early Earth conditions of rapid heat loss after core formation, SiO$_2$ diapirs of ~1 km diameter could have risen independently of mantle flow to their level of neutral buoyancy in the mantle, trapping them there due to a combination of intrinsically high viscosity and neutral buoyancy. We examine the SiO$_2$ yield by assuming Si+O saturation at the conditions found at the base of a magma ocean and find that for a range of conditions, dispersed bodies could reach as high as 8.5 vol.% in parts of the lower mantle. At such low concentration, their effect on aggregate seismic wavespeeds is within observational seismology uncertainty. However, their presence can account for small-scale scattering in the lower mantle due to the bodies' large velocity contrast. We conclude that the shallow lower mantle (700-1500 km depth) could harbor SiO$_2$ released in early Earth times.