The Square Kilometre Array (SKA) will be both the largest radio telescope ever constructed and the largest Big Data project in the known Universe. The first phase of the project will generate on the order of 5 zettabytes of data per year. A critical task for the SKA will be its ability to process data for science, which will need to be conducted by science pipelines. Together with polarization data from the LOFAR Multifrequency Snapshot Sky Survey (MSSS), we have been developing a realistic SKA-like science pipeline that can handle the large data volumes generated by LOFAR at 150 MHz. The pipeline uses task-based parallelism to image, detect sources, and perform Faraday Tomography across the entire LOFAR sky. The project thereby provides a unique opportunity to contribute to the technological development of the SKA telescope, while simultaneously enabling cutting-edge scientific results. In this paper, we provide an update on current efforts to develop a science pipeline that can enable tight constraints on the magnetised large-scale structure of the Universe.

This paper presents a detailed analysis of the applicability and benefits of baseline dependent averaging (BDA) in modern radio interferometers and in particular the Square Kilometre Array (SKA). We demonstrate that BDA does not affect the information content of the data other than a well-defined decorrelation loss for which closed form expressions are readily available. We verify these theoretical findings using simulations. We therefore conclude that BDA can be used reliably in modern radio interferometry allowing a reduction of visibility data volume (and hence processing costs for handling visibility data) by more than 80%.

Fast Radio Bursts (FRBs), are millisecond radio signals that exhibit dispersion larger than what the Galactic electron density can account for. We have conducted a 1446 hour survey for Fast Radio Bursts (FRBs) at 145~MHz, covering a total of 4193 sq. deg on the sky. We used the UK station of the LOFAR radio telescope -- the Rawlings Array -- , accompanied for a majority of the time by the LOFAR station at Nan\c{c}ay, observing the same fields at the same frequency. Our real-time search backend, ARTEMIS, utilizes graphics processing units to search for pulses with dispersion measures up to 320 cm$^{-3}$ pc. Previous derived FRB rates from surveys around 1.4~GHz, and favoured FRB interpretations, motivated this survey, despite all previous detections occurring at higher dispersion measures. We detected no new FRBs above a signal-to-noise threshold of 10, leading to the most stringent upper limit yet on the FRB event rate at these frequencies: 29 sky$^{-1}$ day$^{-1}$ for 5~ms-duration pulses above 62~Jy. The non-detection could be due to scatter-broadening, limitations on the volume and time searched, or the shape of FRB flux density spectra. Assuming the latter and that FRBs are standard candles, the non-detection is compatible with the published FRB sky rate, if their spectra follow a power law with frequency ($\propto \nu^{\alpha}$), with $\alpha\gtrsim+0.1$, demonstrating a marked difference from pulsar spectra. Our results suggest that surveys at higher frequencies, including the low frequency component of the Square Kilometre Array, will have better chances to detect, estimate rates and understand the origin and properties of FRBs.

Context. Modern radio astronomical arrays have (or will have) more than one order of magnitude more receivers than classical synthesis arrays, such as the VLA and the WSRT. This makes gain calibration a computationally demanding task. Several alternating direction implicit (ADI) approaches have therefore been proposed that reduce numerical complexity for this task from $\mathcal{O}(P^3)$ to $\mathcal{O}(P^2)$, where $P$ is the number of receive paths to be calibrated. Aims. We present an ADI method, show that it converges to the optimal solution, and assess its numerical, computational and statistical performance. We also discuss its suitability for application in self-calibration and report on its successful application in LOFAR standard pipelines. Methods. Convergence is proved by rigorous mathematical analysis using a contraction mapping. Its numerical, algorithmic, and statistical performance, as well as its suitability for application in self-calibration, are assessed using simulations. Results. Our simulations confirm the $\mathcal{O}(P^2)$ complexity and excellent numerical and computational properties of the algorithm. They also confirm that the algorithm performs at or close to the Cramer-Rao bound (CRB, lower bound on the variance of estimated parameters). We find that the algorithm is suitable for application in self-calibration and discuss how it can be included. We demonstrate an order-of-magnitude speed improvement in calibration over traditional methods on actual LOFAR data. Conclusions. In this paper, we demonstrate that ADI methods are a valid and computationally more efficient alternative to traditional gain calibration method and we report on its successful application in a number of actual data reduction pipelines.

Astrophysical radio transients are excellent probes of extreme physical processes originating from compact sources within our Galaxy and beyond. Radio frequency signals emitted from these objects provide a means to study the intervening medium through which they travel. Next generation radio telescopes are designed to explore the vast unexplored parameter space of high time resolution astronomy, but require High Performance Computing (HPC) solutions to process the enormous volumes of data that are produced by these telescopes. We have developed a combined software /hardware solution (code named ARTEMIS) for real-time searches for millisecond radio transients, which uses GPU technology to remove interstellar dispersion and detect millisecond radio bursts from astronomical sources in real-time. Here we present an introduction to ARTEMIS. We give a brief overview of the software pipeline, then focus specifically on the intricacies of performing incoherent de-dispersion. We present results from two brute-force algorithms. The first is a GPU based algorithm, designed to exploit the L1 cache of the NVIDIA Fermi GPU. Our second algorithm is CPU based and exploits the new AVX units in Intel Sandy Bridge CPUs.

The identification, and subsequent discovery, of fast radio transients through blind-search surveys requires a large amount of processing power, in worst cases scaling as $\mathcal{O}(N^3)$. For this reason, survey data are generally processed offline, using high-performance computing architectures or hardware-based designs. In recent years, graphics processing units have been extensively used for numerical analysis and scientific simulations, especially after the introduction of new high-level application programming interfaces. Here we show how GPUs can be used for fast transient discovery in real-time. We present a solution to the problem of de-dispersion, providing performance comparisons with a typical computing machine and traditional pulsar processing software. We describe the architecture of a real-time, GPU-based transient search machine. In terms of performance, our GPU solution provides a speed-up factor of between 50 and 200, depending on the parameters of the search.