The astrophysical $S$-factor for the radiative proton capture reaction on $^7$Be ($S_{17}$) at low energies is affected by the $s$-wave scattering lengths. We report the measurement of elastic and inelastic scattering cross sections for the $^7$Be+p system in the center-of-mass energy range 0.474 - 2.740 MeV and center-of-mass angular range of 70$^\circ$- 150$^\circ$. A radioactive $^7$Be beam produced at Oak Ridge National Laboratory's (ORNL) Holifield Radioactive Ion Beam Facility was accelerated and bombarded a thin polypropylene (CH$_{2}$)$_\text n$ target. Scattered ions were detected in the segmented Silicon Detector Array. Using an $\textit{R}$-matrix analysis of ORNL and Louvain-la-Neuve cross section data, the $s$-wave scattering lengths for channel spins 1 and 2 were determined to be 17.34$^{+1.11}_{-1.33}$ and -3.18$^{+0.55}_{-0.50}$ fm, respectively. The uncertainty in the $s$-wave scattering lengths reported in this work is smaller by a factor of 5-8 compared to the previous measurement, which may reduce the overall uncertainty in $S_{17}$ at zero energy. The level structure of $^8$B is discussed based upon the results from this work. Evidence for the existence of 0$^+$ and 2$^+$ levels in $^8$B at 1.9 and 2.21 MeV, respectively, is observed.