GNOSIS is a prototype astrophotonic instrument that utilizes OH suppression fibres consisting of fibre Bragg gratings and photonic lanterns to suppress the 103 brightest atmospheric emission doublets between 1.47-1.7 microns. GNOSIS was commissioned at the 3.9-meter Anglo-Australian Telescope with the IRIS2 spectrograph to demonstrate the potential of OH suppression fibres, but may be potentially used with any telescope and spectrograph combination. Unlike previous atmospheric suppression techniques GNOSIS suppresses the lines before dispersion and in a manner that depends purely on wavelength. We present the instrument design and report the results of laboratory and on-sky tests from commissioning. While these tests demonstrated high throughput and excellent suppression of the skylines by the OH suppression fibres, surprisingly GNOSIS produced no significant reduction in the interline background and the sensitivity of GNOSIS and IRIS2 is about the same as IRIS2. It is unclear whether the lack of reduction in the interline background is due to physical sources or systematic errors as the observations are detector noise-dominated. OH suppression fibres could potentially impact ground-based astronomy at the level of adaptive optics or greater. However, until a clear reduction in the interline background and the corresponding increasing in sensitivity is demonstrated optimized OH suppression fibres paired with a fibre-fed spectrograph will at least provide a real benefits at low resolving powers.

The background noise between 1 and 1.8 microns in ground-based instruments is dominated by atmospheric emission from hydroxyl molecules. We have built and commissioned a new instrument, GNOSIS, which suppresses 103 OH doublets between 1.47 - 1.7 microns by a factor of ~1000 with a resolving power of ~10,000. We present the first results from the commissioning of GNOSIS using the IRIS2 spectrograph at the AAT. The combined throughput of the GNOSIS fore-optics, grating unit and relay optics is ~36 per cent, but this could be improved to ~46 per cent with a more optimal design. We measure strong suppression of the OH lines, confirming that OH suppression with fibre Bragg gratings will be a powerful technology for low resolution spectroscopy. The integrated OH suppressed background between 1.5 and 1.7 microns is reduced by a factor of 9 compared to a control spectrum using the same system without suppression. The potential of low resolution OH suppressed spectroscopy is illustrated with example observations. The GNOSIS background is dominated by detector dark current below 1.67 microns and by thermal emission above 1.67 microns. After subtracting these we detect an unidentified residual interline component of ~ 860 +/ 210 ph/s/m^2/micron/arcsec^2. This component is equally bright in the suppressed and control spectra. We have investigated the possible source of the interline component, but were unable to discriminate between a possible instrumental artifact and intrinsic atmospheric emission. Resolving the source of this emission is crucial for the design of fully optimised OH suppression spectrographs. The next generation OH suppression spectrograph will be focussed on resolving the source of the interline component, taking advantage of better optimisation for a FBG feed. We quantify the necessary improvements for an optimal OH suppressing fibre spectrograph design.