Dark matter coupled solely gravitationally can be produced through the decay of primordial black holes in the early universe. If the dark matter is lighter than the initial black hole temperature, it could be warm enough to be subject to structure formation constraints. In this paper we perform a more precise determination of these constraints. We first evaluate the dark matter phase-space distribution, without relying on the instantaneous decay approximation. We then interface this phase-space distribution with the Boltzmann code CLASS to extract the corresponding matter power spectrum, which we find to match closely those of warm dark matter models, albeit with a different dark matter mass. This mapping allows us to extract constraints from Lyman-$\alpha$ data without the need to perform hydrodynamical simulations. We robustly rule out the possibility, consistent with previous analytic estimates, of primordial black holes having come to dominate the energy density of the universe and simultaneously given rise to all the DM through their decay. Consequences and implications for dark radiation and leptogenesis are also briefly discussed.