We use N-body simulations to explore the origin and a plausible orbit for the Orphan Stream, one of the faintest substructures discovered so far in the outer halo of our Galaxy. We are able to reproduce its position, velocity and distance measurements by appealing to a single wrap of a double-component satellite galaxy. We find that the progenitor of the Orphan Stream could have been an object similar to today's Milky Way dwarfs, such as Carina, Draco, Leo II or Sculptor; and unlikely to be connected to Complex A or Ursa Major II. Our models suggest that such progenitors, if accreted on orbits with apocenters smaller than ~35 kpc, are likely to give rise to very low surface brightness streams, which may be hiding in the outer halo and remain largely undetected with current techniques. The systematic discovery of these ghostly substructures may well require wide field spectroscopic surveys of the Milky Way's outer stellar halo.