We present the first special relativistic, axisymmetric hydrodynamic simulations of black hole-torus systems (approximating general relativistic gravity) as remnants of binary-neutron star (NS-NS) and neutron star-black hole (NS-BH) mergers, in which the viscously driven evolution of the accretion torus is followed with self-consistent energy-dependent neutrino transport and the interaction with the cloud of dynamical ejecta expelled during the NS-NS merging is taken into account. The modeled torus masses, BH masses and spins, and the ejecta masses, velocities, and spatial distributions are adopted from relativistic merger simulations. We find that energy deposition by neutrino annihilation can accelerate outflows with initially high Lorentz factors along polar low-density funnels, but only in mergers with extremely low baryon pollution in the polar regions. NS-BH mergers, where polar mass ejection during the merging phase is absent, provide sufficiently baryon-poor environments to enable neutrino-powered, ultrarelativistic jets with terminal Lorentz factors above 100 and considerable dynamical collimation, favoring short gamma-ray bursts (sGRBs), although their typical energies and durations might be too small to explain the majority of events. In the case of NS-NS mergers, however, neutrino emission of the accreting and viscously spreading torus is too short and too weak to yield enough energy for the outflows to break out from the surrounding ejecta shell as highly relativistic jets. We conclude that neutrino annihilation alone cannot power sGRBs from NS-NS mergers.