The HIFI data processing pipeline was developed to systematically process diagnostic, calibration and astronomical observations taken with the HIFI science instrumentas part of the Herschel mission. The HIFI pipeline processed data from all HIFI observing modes within the Herschel automated processing environment, as well as, within an interactive environment. A common software framework was developed to best support the use cases required by the instrument teams and by the general astronomers. The HIFI pipeline was built on top of that and was designed with a high degree of modularity. This modular design provided the necessary flexibility and extensibility to deal with the complexity of batch-processing eighteen different observing modes, to support the astronomers in the interactive analysis and to cope with adjustments necessary to improve the pipeline and the quality of the end-products. This approach to the software development and data processing effort was arrived at by coalescing the lessons learned from similar research based projects with the understanding that a degree of foresight was required given the overall length of the project. In this article, both the successes and challenges of the HIFI software development process are presented. To support future similar projects and retain experience gained lessons learned are extracted.

{Context}. The HIFI instrument on the Herschel Space Observatory performed over 9100 astronomical observations, almost 900 of which were calibration observations in the course of the nearly four-year Herschel mission. The data from each observation had to be converted from raw telemetry into calibrated products and were included in the Herschel Science Archive. {Aims}. The HIFI pipeline was designed to provide robust conversion from raw telemetry into calibrated data throughout all phases of the HIFI missions. Pre-launch laboratory testing was supported as were routine mission operations. {Methods}. A modular software design allowed components to be easily added, removed, amended and/or extended as the understanding of the HIFI data developed during and after mission operations. {Results}. The HIFI pipeline processed data from all HIFI observing modes within the Herschel automated processing environment as well as within an interactive environment. The same software can be used by the general astronomical community to reprocess any standard HIFI observation. The pipeline also recorded the consistency of processing results and provided automated quality reports. Many pipeline modules were in use since the HIFI pre-launch instrument level testing. {Conclusions}. Processing in steps facilitated data analysis to discover and address instrument artefacts and uncertainties. The availability of the same pipeline components from pre-launch throughout the mission made for well-understood, tested, and stable processing. A smooth transition from one phase to the next significantly enhanced processing reliability and robustness.

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

CCAT will be a 25m diameter sub-millimeter telescope capable of operating in the 0.2 to 2.1mm wavelength range. It will be located at an altitude of 5600m on Cerro Chajnantor in northern Chile near the ALMA site. The anticipated first generation instruments include large format (60,000 pixel) kinetic inductance detector (KID) cameras, a large format heterodyne array and a direct detection multi-object spectrometer. The paper describes the architecture of the CCAT software and the development strategy.

CCAT will be a 25-meter telescope for sub millimeter astronomy located at 5600m altitude on Cerro Chajnantor in northern Chile. CCAT will combine high sensitivity, a wide field of view, and a broad wavelength range (0.35 to 2.1mm) to provide an unprecedented capability for deep, large-area multicolor submillimeter surveys. It is planned to have a suite of instruments including large format KID cameras, a large heterodyne array and a KID-based direct detection multi-object spectrometer. The remote location drives a desire for fully autonomous observing coupled with data reduction pipelines and fast feedback to principal investigators.

In contrast to extensively studied dense star-forming cores, little is known about diffuse gas surrounding star-forming regions. We study molecular gas in the high-mass star-forming region NGC6334I, which contains diffuse, quiescent components that are inconspicuous in widely used molecular tracers such as CO. We present Herschel/HIFI observations of CH toward NGC6334I observed as part of the CHESS key program. HIFI resolves the hyperfine components of its J=3/2-1/2 transition, observed in both emission and absorption. The CH emission appears close to the systemic velocity of NGC6334I, while its measured linewidth of 3 km/s is smaller than previously observed in dense gas tracers such as NH3 and SiO. The CH abundance in the hot core is 7 10^-11, two to three orders of magnitude lower than in diffuse clouds. While other studies find distinct outflows in, e.g., CO and H2O toward NGC6334I, we do not detect outflow signatures in CH. To explain the absorption signatures, at least two absorbing components are needed at -3.0 and +6.5 km/s with N(CH)=7 10^13 and 3 10^13 cm^-2. Two additional absorbing clouds are found at +8.0 and 0.0 km/s, both with N(CH)=2 10^13 cm^-2. Turbulent linewidths for the four absorption components vary between 1.5 and 5.0 km/s in FWHM. We constrain physical properties of our CH clouds by matching our CH absorbers with other absorption signatures. In the hot core, molecules such as H2O and CO trace gas that is heated and dynamically influenced by outflow activity, whereas CH traces more quiescent material. The four CH absorbers have column densities and turbulent properties consistent with diffuse clouds: two are located near NGC6334, and two are unrelated foreground clouds. Local density and dynamical effects influence the chemical composition of physical components of NGC6334, causing some components to be seen in CH but not in other tracers, and vice versa.

Circumstellar envelopes (CSEs) of a variety of evolved stars have been found to contain ammonia (NH3) in amounts that exceed predictions from conventional chemical models by many orders of magnitude. The observations reported here were performed in order to better constrain the NH3 abundance in the CSEs of four, quite diverse, oxygen-rich stars using the NH3 ortho J_K = 1_0 - 0_0 ground-state line. We used the Heterodyne Instrument for the Far Infrared aboard Herschel to observe the NH3 J_K = 1_0 - 0_0 transition near 572.5 GHz, simultaneously with the ortho-H2O J_Ka,Kc = 1_1,0 -1_0,1 transition, toward VY CMa, OH 26.5+0.6, IRC+10420, and IK Tau. We conducted non-LTE radiative transfer modeling with the goal to derive the NH3 abundance in these objects' CSEs. For the latter two stars, Very Large Array imaging of NH3 radio-wavelength inversion lines were used to provide further constraints, particularly on the spatial extent of the NH3-emitting regions. Results. We find remarkably strong NH3 emission in all of our objects with the NH3 line intensities rivaling those obtained for the ground state H2O line. The NH3 abundances relative to H2 are very high and range from 2 x 10-7 to 3 x 10-6 for the objects we have studied. Our observations confirm and even deepen the circumstellar NH3 enigma. While our radiative transfer modeling does not yield satisfactory fits to the observed line profiles, it leads to abundance estimates that confirm the very high values found in earlier studies. New ways to tackle this mystery will include further Herschel observations of more NH3 lines and imaging with the Expanded Very Large Array.

We report the discovery of water vapour toward the carbon star V Cygni. We have used Herschel's HIFI instrument, in dual beam switch mode, to observe the 1(11) - 0(00) para-water transition at 1113.3430 GHz in the upper sideband of the Band 4b receiver. The observed spectral line profile is nearly parabolic, but with a slight asymmetry associated with blueshifted absorption, and the integrated antenna temperature is 1.69 \pm 0.17 K km/s. This detection of thermal water vapour emission, carried out as part of a small survey of water in carbon-rich stars, is only the second such detection toward a carbon-rich AGB star, the first having been obtained by the Submillimeter Wave Astronomy Satellite toward IRC+10216. For an assumed ortho-to-para ratio of 3 for water, the observed line intensity implies a water outflow rate ~ (3 - 6) E-5 Earth masses per year and a water abundance relative to H2 of ~ (2-5) E-6. This value is a factor of at least 1E+4 larger than the expected photospheric abundance in a carbon-rich environment, and - as in IRC+10216 - raises the intriguing possibility that the observed water is produced by the vapourisation of orbiting comets or dwarf planets. However, observations of the single line observed to date do not permit us to place strong constraints upon the spatial distribution or origin of the observed water, but future observations of additional transitions will allow us to determine the inner radius of the H2O-emitting zone, and the H2O ortho-to-para ratio, and thereby to place important constraints upon the origin of the observed water emission.