SURF is the collaborative ICT organisation for Dutch education & research and coordinates the national e-infrastructure (surf.nl). To accelerate scientific discovery, SURF invests in, operates and explores high-end IT solutions for and with researchers in the Netherlands. In this ADASS 2019 contribution we present our latest developments in high performance and high throughput cloud computing. These developments are particularly relevant for Astronomy, as this science domain: (i) generates large (Petabyte sized) data collections, (ii) uses rapid release and deployment schemes for their software, (iii) requires flexible and interactive test and staging environments and (iv) needs to execute complex workflows on diverse data structures. We highlight our new OpenStack-based, cloud infrastructure layer and focus on Spider, our new addition to SURF's high throughput data processing platforms.

The new generation of high-resolution broad-band radio telescopes, like the Low Frequency Array (LOFAR), produces, depending on the level of compression, between 1 to 10 TB of data per hour after correlation. Such a large amount of scientific data demand powerful computing resources and efficient data handling strategies to be mastered. The LOFAR Two-metre Sky Survey (LoTSS) is a Key Science Project (KSP) of the LOFAR telescope. It aims to map the entire northern hemisphere at unprecedented sensitivity and resolution. The survey consist of 3 168 pointings, requiring about 30 PBytes of storage space. As a member of the German Long Wavelength Consortioum (GLOW) the Forschungszentrum J\"ulich (FSZ) stores in the Long Term Archive (LTA) about 50% of all LoTSS observations conducted to date. In collaboration with SURFsara in Amsterdam we developed service tools that enable the KSP to process LOFAR data stored in the J\"ulich LTA at the supercomputer JUWELS in an automated and robust fashion. Through our system more than 500 out of 800 existing LoTSS observations have already been processed with the prefactor pipeline. This pipeline calibrates the direction-independent instrumental and ionospheric effects and furthermore reduces the data size significantly. For continuum imaging, this processing pipeline is the standard pipeline that is executed before more advanced processing and image reconstruction methods are applied.

The LOFAR Two-metre Sky Survey (LoTSS) is an ongoing sensitive, high-resolution 120-168MHz survey of the entire northern sky for which observations are now 20% complete. We present our first full-quality public data release. For this data release 424 square degrees, or 2% of the eventual coverage, in the region of the HETDEX Spring Field (right ascension 10h45m00s to 15h30m00s and declination 45$^\circ$00$'$00$''$ to 57$^\circ$00$'$00$''$) were mapped using a fully automated direction-dependent calibration and imaging pipeline that we developed. A total of 325,694 sources are detected with a signal of at least five times the noise, and the source density is a factor of $\sim 10$ higher than the most sensitive existing very wide-area radio-continuum surveys. The median sensitivity is S$_{\rm 144 MHz} = 71\,\mu$Jy beam$^{-1}$ and the point-source completeness is 90% at an integrated flux density of 0.45mJy. The resolution of the images is 6$''$ and the positional accuracy is within 0.2$''$. This data release consists of a catalogue containing location, flux, and shape estimates together with 58 mosaic images that cover the catalogued area. In this paper we provide an overview of the data release with a focus on the processing of the LOFAR data and the characteristics of the resulting images. In two accompanying papers we provide the radio source associations and deblending and, where possible, the optical identifications of the radio sources together with the photometric redshifts and properties of the host galaxies. These data release papers are published together with a further $\sim$20 articles that highlight the scientific potential of LoTSS.

The Low Frequency Array (LOFAR) radio telescope is an international aperture synthesis radio telescope used to study the Universe at low frequencies. One of the goals of the LOFAR telescope is to conduct deep wide-field surveys. Here we will discuss a framework for the processing of the LOFAR Two Meter Sky Survey (LoTSS). This survey will produce close to 50 PB of data within five years. These data rates require processing at locations with high-speed access to the archived data. To complete the LoTSS project, the processing software needs to be made portable and moved to clusters with a high bandwidth connection to the data archive. This work presents a framework that makes the LOFAR software portable, and is used to scale out LOFAR data reduction. Previous work was successful in preprocessing LOFAR data on a cluster of isolated nodes. This framework builds upon it and and is currently operational. It is designed to be portable, scalable, automated and general. This paper describes its design and high level operation and the initial results processing LoTSS data.

We have conducted two pilot surveys for radio pulsars and fast transients with the Low-Frequency Array (LOFAR) around 140 MHz and here report on the first low-frequency fast-radio burst limit and the discovery of two new pulsars. The first survey, the LOFAR Pilot Pulsar Survey (LPPS), observed a large fraction of the northern sky, ~1.4 x 10^4 sq. deg, with 1-hr dwell times. Each observation covered ~75 sq. deg using 7 independent fields formed by incoherently summing the high-band antenna fields. The second pilot survey, the LOFAR Tied-Array Survey (LOTAS), spanned ~600 sq. deg, with roughly a 5-fold increase in sensitivity compared with LPPS. Using a coherent sum of the 6 LOFAR "Superterp" stations, we formed 19 tied-array beams, together covering 4 sq. deg per pointing. From LPPS we derive a limit on the occurrence, at 142 MHz, of dispersed radio bursts of < 150 /day/sky, for bursts brighter than S > 107 Jy for the narrowest searched burst duration of 0.66 ms. In LPPS, we re-detected 65 previously known pulsars. LOTAS discovered two pulsars, the first with LOFAR or any digital aperture array. LOTAS also re-detected 27 previously known pulsars. These pilot studies show that LOFAR can efficiently carry out all-sky surveys for pulsars and fast transients, and they set the stage for further surveying efforts using LOFAR and the planned low-frequency component of the Square Kilometer Array.