We analyze the early-time isotropic cosmology in the so-called energy-momentum-squared gravity (EMSG). In this theory, a $T_{\mu\nu}T^{\mu\nu}$ term is added to the Einstein-Hilbert action, which has been shown to replace the initial singularity by a regular bounce. We show that this is not the case, and the bouncing solution obtained does not describe our Universe since it belongs to a different solution branch. The solution branch that corresponds to our Universe, while nonsingular, is geodesically incomplete. We analyze the conditions for having viable regular-bouncing solutions in a general class of theories that modify gravity by adding higher order matter terms. Applying these conditions on generalizations of EMSG that add a $\left(T_{\mu\nu}T^{\mu\nu}\right)^{n}$ term to the action, we show that the case of $n=5/8$ is the only one that can give a viable bouncing solution, while the $n>5/8$ cases suffer from the same problem as EMSG, i.e. they give nonsingular, geodesically incomplete solutions. Furthermore, we show that the $1/2<n<5/8$ cases can provide a nonsingular initially de Sitter solution. Finally, the expanding, geodesically incomplete branch of EMSG or its generalizations can be combined with its contracting counterpart using junction conditions to provide a (weakly) singular bouncing solution. We outline the junction conditions needed for this extension and provide the extended solution explicitly for EMSG. In this sense, EMSG replaces the standard early-time singularity by a singular bounce instead of a regular one.