Mitigation of radio frequency interference (RFI) is essential to deliver science-ready radio interferometric data to astronomers. In this paper, using dual polarized radio interferometers, we propose to use the polarization information of post-correlation interference signals to detect and mitigate them. We use the directional statistics of the polarized signals as the detection criteria and formulate a distributed, wideband spectrum sensing problem. Using consensus optimization, we solve this in an online manner, working with mini-batches of data. We present extensive results based on simulations to demonstrate the feasibility of our method.

With ever increasing data rates produced by modern radio telescopes like LOFAR and future telescopes like the SKA, many data processing steps are overwhelmed by the amount of data that needs to be handled using limited compute resources. Calibration is one such operation that dominates the overall data processing computational cost, nonetheless, it is an essential operation to reach many science goals. Calibration algorithms do exist that scale well with the number of stations of an array and the number of directions being calibrated. However, the remaining bottleneck is the raw data volume, which scales with the number of baselines, and which is proportional to the square of the number of stations. We propose a 'stochastic' calibration strategy where we only read in a mini-batch of data for obtaining calibration solutions, as opposed to reading the full batch of data being calibrated. Nonetheless, we obtain solutions that are valid for the full batch of data. Normally, data need to be averaged before calibration is performed to accommodate the data in size-limited compute memory. Stochastic calibration overcomes the need for data averaging before any calibration can be performed, and offers many advantages including: enabling the mitigation of faint radio frequency interference; better removal of strong celestial sources from the data; and better detection and spatial localization of fast radio transients.

A new upper limit on the 21-cm signal power spectrum at a redshift of $z \approx 9.1$ is presented, based on 141 hours of data obtained with the Low-Frequency Array (LOFAR). The analysis includes significant improvements in spectrally-smooth gain-calibration, Gaussian Process Regression (GPR) foreground mitigation and optimally-weighted power spectrum inference. Previously seen `excess power' due to spectral structure in the gain solutions has markedly reduced but some excess power still remains with a spectral correlation distinct from thermal noise. This excess has a spectral coherence scale of $0.25 - 0.45$\,MHz and is partially correlated between nights, especially in the foreground wedge region. The correlation is stronger between nights covering similar local sidereal times. A best 2-$\sigma$ upper limit of $\Delta^2_{21} < (73)^2\,\mathrm{mK^2}$ at $k = 0.075\,\mathrm{h\,cMpc^{-1}}$ is found, an improvement by a factor $\approx 8$ in power compared to the previously reported upper limit. The remaining excess power could be due to residual foreground emission from sources or diffuse emission far away from the phase centre, polarization leakage, chromatic calibration errors, ionosphere, or low-level radio-frequency interference. We discuss future improvements to the signal processing chain that can further reduce or even eliminate these causes of excess power.

We derive constraints on the thermal and ionization states of the intergalactic medium (IGM) at redshift $\approx$ 9.1 using new upper limits on the 21-cm power spectrum measured by the LOFAR radio-telescope and a prior on the ionized fraction at that redshift estimated from recent cosmic microwave background (CMB) observations. We have used results from the reionization simulation code GRIZZLY and a Bayesian inference framework to constrain the parameters which describe the physical state of the IGM. We find that, if the gas heating remains negligible, an IGM with ionized fraction $\gtrsim 0.13$ and a distribution of the ionized regions with a characteristic size $\gtrsim 8 ~h^{-1}$ comoving megaparsec (Mpc) and a full width at the half maximum (FWHM) $\gtrsim 16 ~h^{-1}$ Mpc is ruled out. For an IGM with a uniform spin temperature $T_{\rm S} \gtrsim 3$ K, no constraints on the ionized component can be computed. If the large-scale fluctuations of the signal are driven by spin temperature fluctuations, an IGM with a volume fraction $\lesssim 0.34$ of heated regions with a temperature larger than CMB, average gas temperature 7-160 K and a distribution of the heated regions with characteristic size 3.5-70 $h^{-1}$ Mpc and FWHM of $\lesssim 110$ $h^{-1}$ Mpc is ruled out. These constraints are within the 95 per cent credible intervals. With more stringent future upper limits from LOFAR at multiple redshifts, the constraints will become tighter and will exclude an increasingly large region of the parameter space.

SAGECal has been designed to find the most accurate calibration solutions for low radio frequency imaging observations, with minimum artefacts due to incomplete sky models. SAGECAL is developed to handle extremely large datasets, e.g., when the number of frequency bands greatly exceeds the number of available nodes on a compute cluster. Accurate calibration solutions are derived at the expense of large computational loads, which require distributed computing and modern compute devices, such as GPUs, to decrease runtimes. In this work, we investigate if the GPU version of SAGECal scales well enough to meet the requirements for the Square Kilometre Array and we compare its performance with the CPU version.

Observations of the redshifted 21-cm hyperfine line of neutral hydrogen from early phases of the Universe such as Cosmic Dawn and the Epoch of Reionization promise to open a new window onto the early formation of stars and galaxies. We present the first upper limits on the power spectrum of redshifted 21-cm brightness temperature fluctuations in the redshift range $z = 19.8 - 25.2$ ($54-68$ MHz frequency range) using 14 hours of data obtained with the LOFAR-Low Band Antenna (LBA) array. We also demonstrate the application of a multiple pointing calibration technique to calibrate the LOFAR-LBA dual-pointing observations centred on the North Celestial Pole and the radio galaxy 3C220.3. We observe an unexplained excess of $\sim 30-50\%$ in Stokes $I$ noise compared to Stokes $V$ for the two observed fields, which decorrelates on $\gtrsim 12$ seconds and might have a physical origin. We show that enforcing smoothness of gain errors along frequency direction during calibration reduces the additional variance in Stokes $I$ compared Stokes $V$ introduced by the calibration on sub-band level. After subtraction of smooth foregrounds, we achieve a $2\sigma$ upper limit on the 21-cm power spectrum of $\Delta_{21}^2 < (14561\,\text{mK})^2$ at $k\sim 0.038\,h\,\text{cMpc}^{-1}$ and $\Delta_{21}^2 < (14886\,\text{mK})^2$ at $k\sim 0.038 \,h\,\text{cMpc}^{-1}$ for the 3C220 and NCP fields respectively and both upper limits are consistent with each other. The upper limits for the two fields are still dominated by systematics on most $k$ modes.

Calibration is an essential step in radio interferometric data processing that corrects the data for systematic errors and in addition, subtracts bright foreground interference to reveal weak signals hidden in the residual. These weak and unknown signals are much sought after to reach many science goals but the effect of calibration on such signals is an ever present concern. The main reason for this is the incompleteness of the model used in calibration. Distributed calibration based on consensus optimization has been shown to mitigate the effect due to model incompleteness by calibrating data covering a wide bandwidth in a computationally efficient manner. In this paper, we study the statistical performance of direction dependent distributed calibration, i.e., the distortion caused by calibration on the residual statistics. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship for the influence of the input on the residual and use this to find the relationship between the input and output probability density functions. Using simulations we show that the smallest eigenvalue of the Jacobian of this mapping is a reliable indicator of the statistical performance of calibration. The analysis developed in this paper can also be applied to other data processing steps in radio interferometry such as imaging and foreground subtraction as well as to many other machine learning problems.

We present a stochastic, limited-memory Broyden Fletcher Goldfarb Shanno (LBFGS) algorithm that is suitable for handling very large amounts of data. A direct application of this algorithm is radio interferometric calibration of raw data at fine time and frequency resolution. Almost all existing radio interferometric calibration algorithms assume that it is possible to fit the dataset being calibrated into memory. Therefore, the raw data is averaged in time and frequency to reduce its size by many orders of magnitude before calibration is performed. However, this averaging is detrimental for the detection of some signals of interest that have narrow bandwidth and time duration such as fast radio bursts (FRBs). Using the proposed algorithm, it is possible to calibrate data at such a fine resolution that they cannot be entirely loaded into memory, thus preserving such signals. As an additional demonstration, we use the proposed algorithm for training deep neural networks and compare the performance against the mainstream first order optimization algorithms that are used in deep learning.

Distributed calibration based on consensus optimization is a computationally efficient method to calibrate large radio interferometers such as LOFAR and SKA. Calibrating along multiple directions in the sky and removing the bright foreground signal is a crucial step in many science cases in radio interferometry. The residual data contain weak signals of huge scientific interest and of particular concern is the effect of incomplete sky models used in calibration on the residual. In order to study this, we consider the mapping between the input uncalibrated data and the output residual data. We derive an analytical relationship between the input and output probability density functions which can be used to study the performance of calibration.

Contamination due to foregrounds (Galactic and Extra-galactic), calibration errors and ionospheric effects pose major challenges in detection of the cosmic 21 cm signal in various Epoch of Reionization (EoR) experiments. We present the results of a pilot study of a field centered on 3C196 using LOFAR Low Band (56-70 MHz) observations, where we quantify various wide field and calibration effects such as gain errors, polarized foregrounds, and ionospheric effects. We observe a `pitchfork' structure in the 2D power spectrum of the polarized intensity in delay-baseline space, which leaks into the modes beyond the instrumental horizon (EoR/CD window). We show that this structure largely arises due to strong instrumental polarization leakage ($\sim30\%$) towards {Cas\,A} ($\sim21$ kJy at 81 MHz, brightest source in northern sky), which is far away from primary field of view. We measure an extremely small ionospheric diffractive scale ($r_{\text{diff}} \approx 430$ m at 60 MHz) towards {Cas\,A} resembling pure Kolmogorov turbulence compared to $r_{\text{diff}} \sim3 - 20$ km towards zenith at 150 MHz for typical ionospheric conditions. This is one of the smallest diffractive scales ever measured at these frequencies. Our work provides insights in understanding the nature of aforementioned effects and mitigating them in future Cosmic Dawn observations (e.g. with SKA-low and HERA) in the same frequency window.

Calibration of a typical radio interferometric array yields thousands of parameters as solutions. These solutions contain valuable information about the systematic errors in the data (ionosphere and beam shape). This information could be reused in calibration to improve the accuracy and also can be fed into imaging to improve the fidelity. We propose a distributed optimization strategy to construct models for the systematic errors in the data using the calibration solutions. We formulate this as an elastic net regularized distributed optimization problem which we solve using the alternating direction method of multipliers (ADMM) algorithm. We give simulation results to show the feasibility of the proposed distributed model construction scheme.

New and upcoming radio interferometers will produce unprecedented amounts of data that demand extremely powerful computers for processing. This is a limiting factor due to the large computational power and energy costs involved. Such limitations restrict several key data processing steps in radio interferometry. One such step is calibration where systematic errors in the data are determined and corrected. Accurate calibration is an essential component in reaching many scientific goals in radio astronomy and the use of consensus optimization that exploits the continuity of systematic errors across frequency significantly improves calibration accuracy. In order to reach full consensus, data at all frequencies need to be calibrated simultaneously. In the SKA regime, this can become intractable if the available compute agents do not have the resources to process data from all frequency channels simultaneously. In this paper, we propose a multiplexing scheme that is based on the alternating direction method of multipliers (ADMM) with cyclic updates. With this scheme, it is possible to simultaneously calibrate the full dataset using far fewer compute agents than the number of frequencies at which data are available. We give simulation results to show the feasibility of the proposed multiplexing scheme in simultaneously calibrating a full dataset when a limited number of compute agents are available.

Distributed radio interferometric calibration based on consensus optimization has been shown to improve the estimation of systematic errors in radio astronomical observations. The intrinsic continuity of systematic errors across frequency is used by a consensus polynomial to penalize traditional calibration. Consensus is achieved via the use of alternating direction method of multipliers (ADMM) algorithm. In this paper, we extend the existing distributed calibration algorithms to use ADMM with an adaptive penalty parameter update. Compared to a fixed penalty, its adaptive update has been shown to perform better in diverse applications of ADMM. In this paper, we compare two such popular penalty parameter update schemes: residual balance penalty update and spectral penalty update (Barzilai-Borwein). We apply both schemes to distributed radio interferometric calibration and compare their performance against ADMM with a fixed penalty parameter. Simulations show that both methods of adaptive penalty update improve the convergence of ADMM but the spectral penalty parameter update shows more stability.

We present the first limits on the Epoch of Reionization (EoR) 21-cm HI power spectra, in the redshift range $z=7.9-10.6$, using the Low-Frequency Array (LOFAR) High-Band Antenna (HBA). In total 13\,h of data were used from observations centred on the North Celestial Pole (NCP). After subtraction of the sky model and the noise bias, we detect a non-zero $\Delta^2_{\rm I} = (56 \pm 13 {\rm mK})^2$ (1-$\sigma$) excess variance and a best 2-$\sigma$ upper limit of $\Delta^2_{\rm 21} < (79.6 {\rm mK})^2$ at $k=0.053$$h$cMpc$^{-1}$ in the range $z=$9.6-10.6. The excess variance decreases when optimizing the smoothness of the direction- and frequency-dependent gain calibration, and with increasing the completeness of the sky model. It is likely caused by (i) residual side-lobe noise on calibration baselines, (ii) leverage due to non-linear effects, (iii) noise and ionosphere-induced gain errors, or a combination thereof. Further analyses of the excess variance will be discussed in forthcoming publications.

(abridged). We outline LBCS (the LOFAR Long-Baseline Calibrator Survey), whose aim is to identify sources suitable for calibrating the highest-resolution observations made with the International LOFAR Telescope, which include baselines >1000 km. Suitable sources must contain significant correlated flux density (50-100mJy) at frequencies around 110--190~MHz on scales of a few hundred mas. At least for the 200--300-km international baselines, we find around 1 suitable calibrator source per square degree over a large part of the northern sky, in agreement with previous work. This should allow a randomly selected target to be successfully phase calibrated on the international baselines in over 50% of cases. Products of the survey include calibrator source lists and fringe-rate and delay maps of wide areas -- typically a few degrees -- around each source. The density of sources with significant correlated flux declines noticeably with baseline length over the range 200--600~km, with good calibrators on the longest baselines appearing only at the rate of 0.5 per square degree. Coherence times decrease from 1--3 minutes on 200-km baselines to about 1 minute on 600-km baselines, suggesting that ionospheric phase variations contain components with scales of a few hundred kilometres. The longest median coherence time, at just over 3 minutes, is seen on the DE609 baseline, which at 227km is close to being the shortest. We see median coherence times of between 80 and 110 seconds on the four longest baselines (580--600~km), and about 2 minutes for the other baselines. The success of phase transfer from calibrator to target is shown to be influenced by distance, in a manner that suggests a coherence patch at 150-MHz of the order of 1 degree.

Leakage of diffuse polarized emission into Stokes I caused by the polarized primary beam of the instrument might mimic the spectral structure of the 21-cm signal coming from the epoch of reionization (EoR) making their separation difficult. Therefore, understanding polarimetric performance of the antenna is crucial for a successful detection of the EoR signal. Here, we have calculated the accuracy of the nominal model beam of LOFAR in predicting the leakage from Stokes I to Q, U by comparing them with the corresponding leakage of compact sources actually observed in the 3C295 field. We have found that the model beam has errors of less than or equal to 10% on the predicted levels of leakage of ~1% within the field of view, i. e. if the leakage is taken out perfectly using this model the leakage will reduce to $10^{-3}$ of the Stokes I flux. If similar levels of accuracy can be obtained in removing leakage from Stokes Q, U to I, we can say, based on the results of our previous paper, that the removal of this leakage using this beam model would ensure that the leakage is well below the expected EoR signal in almost the whole instrumental k-space of the cylindrical power spectrum. We have also shown here that direction dependent calibration can remove instrumentally polarized compact sources, given an unpolarized sky model, very close to the local noise level.

LOFAR is the LOw Frequency Radio interferometer ARray located at mid-latitude ($52^{\circ} 53'N$). Here, we present results on ionospheric structures derived from 29 LOFAR nighttime observations during the winters of 2012/2013 and 2013/2014. We show that LOFAR is able to determine differential ionospheric TEC values with an accuracy better than 1 mTECU over distances ranging between 1 and 100 km. For all observations the power law behavior of the phase structure function is confirmed over a long range of baseline lengths, between $1$ and $80$ km, with a slope that is in general larger than the $5/3$ expected for pure Kolmogorov turbulence. The measured average slope is $1.89$ with a one standard deviation spread of $0.1$. The diffractive scale, i.e. the length scale where the phase variance is $1\, \mathrm{rad^2}$, is shown to be an easily obtained single number that represents the ionospheric quality of a radio interferometric observation. A small diffractive scale is equivalent to high phase variability over the field of view as well as a short time coherence of the signal, which limits calibration and imaging quality. For the studied observations the diffractive scales at $150$ MHz vary between $3.5$ and $30\,$ km. A diffractive scale above $5$ km, pertinent to about $90 \%$ of the observations, is considered sufficient for the high dynamic range imaging needed for the LOFAR Epoch of Reionization project. For most nights the ionospheric irregularities were anisotropic, with the structures being aligned with the Earth magnetic field in about $60\%$ of the observations.

We recently proposed the use of consensus optimization as a viable and effective way to improve the quality of calibration of radio interferometric data. We showed that it is possible to obtain far more accurate calibration solutions and also to distribute the compute load across a network of computers by using this technique. A crucial aspect in any consensus optimization problem is the selection of the penalty parameter used in the alternating direction method of multipliers (ADMM) iterations. This affects the convergence speed as well as the accuracy. In this paper, we use the Hessian of the cost function used in calibration to appropriately select this penalty. We extend our results to a multi-directional calibration setting, where we propose to use a penalty scaled by the squared intensity of each direction.

The redshifted 21 cm line of neutral hydrogen is a promising probe of the Epoch of Reionization (EoR). However, its detection requires a thorough understanding and control of the systematic errors. We study two systematic biases observed in the LOFAR EoR residual data after calibration and subtraction of bright discrete foreground sources. The first effect is a suppression in the diffuse foregrounds, which could potentially mean a suppression of the 21 cm signal. The second effect is an excess of noise beyond the thermal noise. The excess noise shows fluctuations on small frequency scales, and hence it can not be easily removed by foreground removal or avoidance methods. Our analysis suggests that sidelobes of residual sources due to the chromatic point spread function and ionospheric scintillation can not be the dominant causes of the excess noise. Rather, both the suppression of diffuse foregrounds and the excess noise can occur due to calibration with an incomplete sky model containing predominantly bright discrete sources. We show that calibrating only on bright sources can cause suppression of other signals and introduce an excess noise in the data. The levels of the suppression and excess noise depend on the relative flux of sources which are not included in the model with respect to the flux of modeled sources. We discuss possible solutions such as using only long baselines to calibrate the interferometric gain solutions as well as simultaneous multi-frequency calibration along with their benefits and shortcomings.

Cosmic rays are the highest energy particles found in nature. Measurements of the mass composition of cosmic rays between 10^{17} eV and 10^{18} eV are essential to understand whether this energy range is dominated by Galactic or extragalactic sources. It has also been proposed that the astrophysical neutrino signal comes from accelerators capable of producing cosmic rays of these energies. Cosmic rays initiate cascades of secondary particles (air showers) in the atmosphere and their masses are inferred from measurements of the atmospheric depth of the shower maximum, Xmax, or the composition of shower particles reaching the ground. Current measurements suffer from either low precision, or a low duty cycle and a high energy threshold. Radio detection of cosmic rays is a rapidly developing technique, suitable for determination of Xmax with a duty cycle of in principle nearly 100%. The radiation is generated by the separation of relativistic charged particles in the geomagnetic field and a negative charge excess in the shower front. Here we report radio measurements of Xmax with a mean precision of 16 g/cm^2 between 10^{17}-10^{17.5} eV. Because of the high resolution in $Xmax we can determine the mass spectrum and find a mixed composition, containing a light mass fraction of ~80%. Unless the extragalactic component becomes significant already below 10^{17.5} eV, our measurements indicate an additional Galactic component dominating at this energy range.

We present the results of a four-month campaign searching for low-frequency radio transients near the North Celestial Pole with the Low-Frequency Array (LOFAR), as part of the Multifrequency Snapshot Sky Survey (MSSS). The data were recorded between 2011 December and 2012 April and comprised 2149 11-minute snapshots, each covering 175 deg^2. We have found one convincing candidate astrophysical transient, with a duration of a few minutes and a flux density at 60 MHz of 15-25 Jy. The transient does not repeat and has no obvious optical or high-energy counterpart, as a result of which its nature is unclear. The detection of this event implies a transient rate at 60 MHz of 3.9 (+14.7, -3.7) x 10^-4 day^-1 deg^-2, and a transient surface density of 1.5 x 10^-5 deg^-2, at a 7.9-Jy limiting flux density and ~10-minute time-scale. The campaign data were also searched for transients at a range of other time-scales, from 0.5 to 297 min, which allowed us to place a range of limits on transient rates at 60 MHz as a function of observation duration.

The existence of double-double radio galaxies (DDRGs) is evidence for recurrent jet activity in AGN, as expected from standard accretion models. A detailed study of these rare sources provides new perspectives for investigating the AGN duty cycle, AGN-galaxy feedback, and accretion mechanisms. Large catalogues of radio sources provide statistical information about the evolution of the radio-loud AGN population out to high redshifts. Using wide-field imaging with the LOFAR telescope, we study both a well-known DDRG as well as a large number of radio sources in the field of view. We present a high resolution image of the DDRG B1834+620 obtained at 144 MHz using LOFAR commissioning data. Our image covers about 100 square degrees and contains over 1000 sources. The four components of the DDRG B1834+620 have been resolved for the first time at 144 MHz. Inner lobes were found to point towards the direction of the outer lobes, unlike standard FR~II sources. Polarized emission was detected in the northern outer lobe. The high spatial resolution allows the identification of a large number of small double-lobed radio sources; roughly 10% of all sources in the field are doubles with a separation smaller than 1 arcmin. The spectral fit of the four components is consistent with a scenario in which the outer lobes are still active or the jets recently switched off, while emission of the inner lobes is the result of a mix-up of new and old jet activity. From the presence of the newly extended features in the inner lobes of the DDRG, we can infer that the mechanism responsible for their formation is the bow shock that is driven by the newly launched jet. We find that the density of the small doubles exceeds the density of FR-II sources with similar properties at 1.4 GHz, but this difference becomes smaller for low flux densities.

This study aims to characterize linear polarization structures in LOFAR observations of the interstellar medium (ISM) in the 3C196 field, one of the primary fields of the LOFAR-Epoch of Reionization key science project. We have used the high band antennas (HBA) of LOFAR to image this region and Rotation Measure (RM) synthesis to unravel the distribution of polarized structures in Faraday depth. The brightness temperature of the detected Galactic emission is $5-15~{\rm K}$ in polarized intensity and covers the range from -3 to +8 ${\rm rad~m^{-2}}$ in Faraday depth. The most interesting morphological feature is a strikingly straight filament at a Faraday depth of $+0.5~{\rm rad~m^{-2}}$ running from north to south, right through the centre of the field and parallel to the Galactic plane. There is also an interesting system of linear depolarization canals conspicuous in an image showing the peaks of Faraday spectra. We used the Westerbork Synthesis Radio Telescope (WSRT) at 350 MHz to image the same region. For the first time, we see some common morphology in the RM cubes made at 150 and 350~{\rm MHz}. There is no indication of diffuse emission in total intensity in the interferometric data, in line with results at higher frequencies and previous LOFAR observations. Based on our results, we determined physical parameters of the ISM and proposed a simple model that may explain the observed distribution of the intervening magneto-ionic medium. The mean line-of-sight magnetic field component, $B_\parallel$, is determined to be $0.3\pm0.1~{\rm \mu G}$ and its spatial variation across the 3C196 field is $0.1~{\rm \mu G}$. The filamentary structure is probably an ionized filament in the ISM, located somewhere within the Local Bubble. This filamentary structure shows an excess in thermal electron density ($n_e B_\parallel>6.2~{\rm cm^{-3}\mu G}$) compared to its surroundings.

We present the Multifrequency Snapshot Sky Survey (MSSS), the first northern-sky LOFAR imaging survey. In this introductory paper, we first describe in detail the motivation and design of the survey. Compared to previous radio surveys, MSSS is exceptional due to its intrinsic multifrequency nature providing information about the spectral properties of the detected sources over more than two octaves (from 30 to 160 MHz). The broadband frequency coverage, together with the fast survey speed generated by LOFAR's multibeaming capabilities, make MSSS the first survey of the sort anticipated to be carried out with the forthcoming Square Kilometre Array (SKA). Two of the sixteen frequency bands included in the survey were chosen to exactly overlap the frequency coverage of large-area Very Large Array (VLA) and Giant Metrewave Radio Telescope (GMRT) surveys at 74 MHz and 151 MHz respectively. The survey performance is illustrated within the "MSSS Verification Field" (MVF), a region of 100 square degrees centered at J2000 (RA,Dec)=(15h,69deg). The MSSS results from the MVF are compared with previous radio survey catalogs. We assess the flux and astrometric uncertainties in the catalog, as well as the completeness and reliability considering our source finding strategy. We determine the 90% completeness levels within the MVF to be 100 mJy at 135 MHz with 108" resolution, and 550 mJy at 50 MHz with 166" resolution. Images and catalogs for the full survey, expected to contain 150,000-200,000 sources, will be released to a public web server. We outline the plans for the ongoing production of the final survey products, and the ultimate public release of images and source catalogs.

We investigate the feasibility of detecting 21cm absorption features in the afterglow spectra of high redshift long Gamma Ray Bursts (GRBs). This is done employing simulations of cosmic reionization, together with the instrumental characteristics of the LOw Frequency ARray (LOFAR). We find that absorption features could be marginally (with a S/N larger than a few) detected by LOFAR at z>7 if the GRB originated from PopIII stars, while the detection would be easier if the noise were reduced by one order of magnitude, i.e. similar to what is expected for the first phase of the Square Kilometer Array (SKA1-low). On the other hand, more standard GRBs are too dim to be detected even with ten times the sensitivity of SKA1-low, and only in the most optimistic case can a S/N larger than a few be reached at z>9.

Detection of the 21-cm signal coming from the epoch of reionization (EoR) is challenging especially because, even after removing the foregrounds, the residual Stokes $I$ maps contain leakage from polarized emission that can mimic the signal. Here, we discuss the instrumental polarization of LOFAR and present realistic simulations of the leakages between Stokes parameters. From the LOFAR observations of polarized emission in the 3C196 field, we have quantified the level of polarization leakage caused by the nominal model beam of LOFAR, and compared it with the EoR signal using power spectrum analysis. We found that at 134--166 MHz, within the central 4$^\circ$ of the field the $(Q,U)\rightarrow I$ leakage power is lower than the EoR signal at $k<0.3$ Mpc$^{-1}$. The leakage was found to be localized around a Faraday depth of 0, and the rms of the leakage as a fraction of the rms of the polarized emission was shown to vary between 0.2-0.3\%, both of which could be utilized in the removal of leakage. Moreover, we could define an `EoR window' in terms of the polarization leakage in the cylindrical power spectrum above the PSF-induced wedge and below $k_\parallel\sim 0.5$ Mpc$^{-1}$, and the window extended up to $k_\parallel\sim 1$ Mpc$^{-1}$ at all $k_\perp$ when 70\% of the leakage had been removed. These LOFAR results show that even a modest polarimetric calibration over a field of view of $\lesssim 4^\circ$ in the future arrays like SKA will ensure that the polarization leakage remains well below the expected EoR signal at the scales of 0.02-1 Mpc$^{-1}$.

PSR B0823+26, a 0.53-s radio pulsar, displays a host of emission phenomena over timescales of seconds to (at least) hours, including nulling, subpulse drifting, and mode-changing. Studying pulsars like PSR B0823+26 provides further insight into the relationship between these various emission phenomena and what they might teach us about pulsar magnetospheres. Here we report on the LOFAR discovery that PSR B0823+26 has a weak and sporadically emitting 'quiet' (Q) emission mode that is over 100 times weaker (on average) and has a nulling fraction forty-times greater than that of the more regularly-emitting 'bright' (B) mode. Previously, the pulsar has been undetected in the Q-mode, and was assumed to be nulling continuously. PSR B0823+26 shows a further decrease in average flux just before the transition into the B-mode, and perhaps truly turns off completely at these times. Furthermore, simultaneous observations taken with the LOFAR, Westerbork, Lovell, and Effelsberg telescopes between 110 MHz and 2.7 GHz demonstrate that the transition between the Q-mode and B-mode occurs within one single rotation of the neutron star, and that it is concurrent across the range of frequencies observed.

Increasing data volumes delivered by a new generation of radio interferometers require computationally efficient and robust calibration algorithms. In this paper, we propose distributed calibration as a way of improving both computational cost as well as robustness in calibration. We exploit the data parallelism across frequency that is inherent in radio astronomical observations that are recorded as multiple channels at different frequencies. Moreover, we also exploit the smoothness of the variation of calibration parameters across frequency. Data parallelism enables us to distribute the computing load across a network of compute agents. Smoothness in frequency enables us reformulate calibration as a consensus optimization problem. With this formulation, we enable flow of information between compute agents calibrating data at different frequencies, without actually passing the data, and thereby improving robustness. We present simulation results to show the feasibility as well as the advantages of distributed calibration as opposed to conventional calibration.

Context. The LOw Frequency ARray (LOFAR) radio telescope is a giant digital phased array interferometer with multiple antennas distributed in Europe. It provides discrete sets of Fourier components of the sky brightness. Recovering the original brightness distribution with aperture synthesis forms an inverse problem that can be solved by various deconvolution and minimization methods Aims. Recent papers have established a clear link between the discrete nature of radio interferometry measurement and the "compressed sensing" (CS) theory, which supports sparse reconstruction methods to form an image from the measured visibilities. Empowered by proximal theory, CS offers a sound framework for efficient global minimization and sparse data representation using fast algorithms. Combined with instrumental direction-dependent effects (DDE) in the scope of a real instrument, we developed and validated a new method based on this framework Methods. We implemented a sparse reconstruction method in the standard LOFAR imaging tool and compared the photometric and resolution performance of this new imager with that of CLEAN-based methods (CLEAN and MS-CLEAN) with simulated and real LOFAR data Results. We show that i) sparse reconstruction performs as well as CLEAN in recovering the flux of point sources; ii) performs much better on extended objects (the root mean square error is reduced by a factor of up to 10); and iii) provides a solution with an effective angular resolution 2-3 times better than the CLEAN images. Conclusions. Sparse recovery gives a correct photometry on high dynamic and wide-field images and improved realistic structures of extended sources (of simulated and real LOFAR datasets). This sparse reconstruction method is compatible with modern interferometric imagers that handle DDE corrections (A- and W-projections) required for current and future instruments such as LOFAR and SKA

Measuring radio emission from air showers offers a novel way to determine properties of the primary cosmic rays such as their mass and energy. Theory predicts that relativistic time compression effects lead to a ring of amplified emission which starts to dominate the emission pattern for frequencies above ~100 MHz. In this article we present the first detailed measurements of this structure. Ring structures in the radio emission of air showers are measured with the LOFAR radio telescope in the frequency range of 110 - 190 MHz. These data are well described by CoREAS simulations. They clearly confirm the importance of including the index of refraction of air as a function of height. Furthermore, the presence of the Cherenkov ring offers the possibility for a geometrical measurement of the depth of shower maximum, which in turn depends on the mass of the primary particle.

Aims. An efficient means of locating calibrator sources for International LOFAR is developed and used to determine the average density of usable calibrator sources on the sky for subarcsecond observations at 140 MHz. Methods. We used the multi-beaming capability of LOFAR to conduct a fast and computationally inexpensive survey with the full International LOFAR array. Sources were pre-selected on the basis of 325 MHz arcminute-scale flux density using existing catalogues. By observing 30 different sources in each of the 12 sets of pointings per hour, we were able to inspect 630 sources in two hours to determine if they possess a sufficiently bright compact component to be usable as LOFAR delay calibrators. Results. Over 40% of the observed sources are detected on multiple baselines between international stations and 86 are classified as satisfactory calibrators. We show that a flat low-frequency spectrum (from 74 to 325 MHz) is the best predictor of compactness at 140 MHz. We extrapolate from our sample to show that the density of calibrators on the sky that are sufficiently bright to calibrate dispersive and non-dispersive delays for the International LOFAR using existing methods is 1.0 per square degree. Conclusions. The observed density of satisfactory delay calibrator sources means that observations with International LOFAR should be possible at virtually any point in the sky, provided that a fast and efficient search using the methodology described here is conducted prior to the observation to identify the best calibrator.

Radio interferometers observe the Fourier space of the sky, at locations determined by the array geometry. Before a real space image is constructed by a Fourier transform, the data is weighted to improve the quality of reconstruction. Two criteria for calculation of weights are maximizing sensitivity and minimizing point spread function (PSF) sidelobe levels. In this paper, we propose a novel weighting scheme suitable for ultra deep imaging experiments. The proposed weighting scheme is used to maximize sensitivity while minimizing PSF sidelobe variation across frequency and multiple epochs. We give simulation results that show the superiority of the proposed scheme compared with commonly used weighting schemes in achieving these objectives.

The Sun is an active source of radio emission which is often associated with energetic phenomena such as solar flares and coronal mass ejections (CMEs). At low radio frequencies (<100 MHz), the Sun has not been imaged extensively because of the instrumental limitations of previous radio telescopes. Here, the combined high spatial, spectral and temporal resolution of the Low Frequency Array (LOFAR) was used to study solar Type III radio bursts at 30-90 MHz and their association with CMEs. The Sun was imaged with 126 simultaneous tied-array beams within 5 solar radii of the solar centre. This method offers benefits over standard interferometric imaging since each beam produces high temporal (83 ms) and spectral resolution (12.5 kHz) dynamic spectra at an array of spatial locations centred on the Sun. LOFAR's standard interferometric output is currently limited to one image per second. Over a period of 30 minutes, multiple Type III radio bursts were observed, a number of which were found to be located at high altitudes (4 solar radii from the solar center at 30 MHz) and to have non-radial trajectories. These bursts occurred at altitudes in excess of values predicted by 1D radial electron density models. The non-radial high altitude Type III bursts were found to be associated with the expanding flank of a CME. The CME may have compressed neighbouring streamer plasma producing larger electron densities at high altitudes, while the non-radial burst trajectories can be explained by the deflection of radial magnetic fields as the CME expanded in the low corona.

We present radio observations of the Moon between $35$ and $80$ MHz to demonstrate a novel technique of interferometrically measuring large-scale diffuse emission extending far beyond the primary beam (global signal) for the first time. In particular, we show that (i) the Moon appears as a negative-flux source at frequencies $35<\nu<80$ MHz since it is `colder' than the diffuse Galactic background it occults, (ii) using the (negative) flux of the lunar disc, we can reconstruct the spectrum of the diffuse Galactic emission with the lunar thermal emission as a reference, and (iii) that reflected RFI (radio-frequency interference) is concentrated at the center of the lunar disc due to specular nature of reflection, and can be independently measured. Our RFI measurements show that (i) Moon-based Cosmic Dawn experiments must design for an Earth-isolation of better than $80$ dB to achieve an RFI temperature $<1$ mK, (ii) Moon-reflected RFI contributes to a dipole temperature less than $20$ mK for Earth-based Cosmic Dawn experiments, (iii) man-made satellite-reflected RFI temperature exceeds $20$ mK if the aggregate cross section of visible satellites exceeds $80$ m$^2$ at $800$ km height, or $5$ m$^2$ at $400$ km height. Currently, our diffuse background spectrum is limited by sidelobe confusion on short baselines (10-15% level). Further refinement of our technique may yield constraints on the redshifted global $21$-cm signal from Cosmic Dawn ($40>z>12$) and the Epoch of Reionization ($12>z>5$).

This study aims to characterise the polarized foreground emission in the ELAIS-N1 field and to address its possible implications for the extraction of the cosmological 21-cm signal from the Low-Frequency Array - Epoch of Reionization (LOFAR-EoR) data. We use the high band antennas of LOFAR to image this region and RM-synthesis to unravel structures of polarized emission at high Galactic latitudes. The brightness temperature of the detected Galactic emission is on average 4 K in polarized intensity and covers the range from -10 to +13rad m^-2 in Faraday depth. The total polarized intensity and polarization angle show a wide range of morphological features. We have also used the Westerbork Synthesis Radio Telescope (WSRT) at 350 MHz to image the same region. The LOFAR and WSRT images show a similar complex morphology, at comparable brightness levels, but their spatial correlation is very low. The fractional polarization at 150 MHz, expressed as a percentage of the total intensity, amounts to 1.5%. There is no indication of diffuse emission in total intensity in the interferometric data, in line with results at higher frequencies. The wide frequency range, good angular resolution and good sensitivity make LOFAR an exquisite instrument for studying Galactic polarized emission at a resolution of 1-2 rad m^-2 in Faraday depth. The different polarised patterns observed at 150 MHz and 350 MHz are consistent with different source distributions along the line of sight wring in a variety of Faraday thin regions of emission. The presence of polarised foregrounds is a serious complication for Epoch of Reionization experiments. To avoid the leakage of polarized emission into total intensity, which can depend on frequency, we need to calibrate the instrumental polarization across the field of view to a small fraction of 1%.

Several experiments are underway to detect the cosmic redshifted 21-cm signal from neutral hydrogen from the Epoch of Reionization (EoR). Due to their very low signal-to-noise ratio, these observations aim for a statistical detection of the signal by measuring its power spectrum. We investigate the extraction of the variance of the signal as a first step towards detecting and constraining the global history of the EoR. Signal variance is the integral of the signal's power spectrum, and it is expected to be measured with a high significance. We demonstrate this through results from a simulation and parameter estimation pipeline developed for the Low Frequency Array (LOFAR)-EoR experiment. We show that LOFAR should be able to detect the EoR in 600 hours of integration using the variance statistic. Additionally, the redshift ($z_r$) and duration ($\Delta z$) of reionization can be constrained assuming a parametrization. We use an EoR simulation of $z_r = 7.68$ and $\Delta z = 0.43$ to test the pipeline. We are able to detect the simulated signal with a significance of 4 standard deviations and extract the EoR parameters as $z_r = 7.72^{+0.37}_{-0.18}$ and $\Delta z = 0.53^{+0.12}_{-0.23}$ in 600 hours, assuming that systematic errors can be adequately controlled. We further show that the significance of detection and constraints on EoR parameters can be improved by measuring the cross-variance of the signal by cross-correlating consecutive redshift bins.

Extensive air showers, induced by high energy cosmic rays impinging on the Earth's atmosphere, produce radio emission that is measured with the LOFAR radio telescope. As the emission comes from a finite distance of a few kilometers, the incident wavefront is non-planar. A spherical, conical or hyperbolic shape of the wavefront has been proposed, but measurements of individual air showers have been inconclusive so far. For a selected high-quality sample of 161 measured extensive air showers, we have reconstructed the wavefront by measuring pulse arrival times to sub-nanosecond precision in 200 to 350 individual antennas. For each measured air shower, we have fitted a conical, spherical, and hyperboloid shape to the arrival times. The fit quality and a likelihood analysis show that a hyperboloid is the best parametrization. Using a non-planar wavefront shape gives an improved angular resolution, when reconstructing the shower arrival direction. Furthermore, a dependence of the wavefront shape on the shower geometry can be seen. This suggests that it will be possible to use a wavefront shape analysis to get an additional handle on the atmospheric depth of the shower maximum, which is sensitive to the mass of the primary particle.

We present the first detection of carbon radio recombination line absorption along the line of sight to Cygnus A. The observations were carried out with the LOw Frequency ARray in the 33 to 57 MHz range. These low frequency radio observations provide us with a new line of sight to study the diffuse, neutral gas in our Galaxy. To our knowledge this is the first time that foreground Milky Way recombination line absorption has been observed against a bright extragalactic background source. By stacking 48 carbon $\alpha$ lines in the observed frequency range we detect carbon absorption with a signal-to-noise ratio of about 5. The average carbon absorption has a peak optical depth of 2$\times$10$^{-4}$, a line width of 10 km s$^{-1}$ and a velocity of +4 km s$^{-1}$ with respect to the local standard of rest. The associated gas is found to have an electron temperature $T_{e}\sim$ 110 K and density $n_{e}\sim$ 0.06 cm$^{-3}$. These properties imply that the observed carbon $\alpha$ absorption likely arises in the cold neutral medium of the Orion arm of the Milky Way. Hydrogen and helium lines were not detected to a 3$\sigma$ peak optical depth limit of 1.5$\times$10$^{-4}$ for a 4 km s$^{-1}$ channel width. Radio recombination lines associated with Cygnus A itself were also searched for, but are not detected. We set a 3$\sigma$ upper limit of 1.5$\times$10$^{-4}$ for the peak optical depth of these lines for a 4 km s$^{-1}$ channel width.

The low frequency array (LOFAR), is the first radio telescope designed with the capability to measure radio emission from cosmic-ray induced air showers in parallel with interferometric observations. In the first $\sim 2\,\mathrm{years}$ of observing, 405 cosmic-ray events in the energy range of $10^{16} - 10^{18}\,\mathrm{eV}$ have been detected in the band from $30 - 80\,\mathrm{MHz}$. Each of these air showers is registered with up to $\sim1000$ independent antennas resulting in measurements of the radio emission with unprecedented detail. This article describes the dataset, as well as the analysis pipeline, and serves as a reference for future papers based on these data. All steps necessary to achieve a full reconstruction of the electric field at every antenna position are explained, including removal of radio frequency interference, correcting for the antenna response and identification of the pulsed signal.

The characteristic outer scale of turbulence and the ratio of the random to ordered components of the magnetic field are key parameters to characterise magnetic turbulence in the interstellar gas, which affects the propagation of cosmic rays within the Galaxy. We provide new constraints to those two parameters. We use the LOw Frequency ARray (LOFAR) to image the diffuse continuum emission in the Fan region at (l,b) (137.0,+7.0) at 80"x70" resolution in the range [146,174] MHz. We detect multi-scale fluctuations in the Galactic synchrotron emission and compute their power spectrum. Applying theoretical estimates and derivations from the literature for the first time, we derive the outer scale of turbulence and the ratio of random to ordered magnetic field from the characteristics of these fluctuations . We obtain the deepest image of the Fan region to date and find diffuse continuum emission within the primary beam. The power spectrum of the foreground synchrotron fluctuations displays a power law behaviour for scales between 100 and 8 arcmin with a slope of (-1.84+/-0.19). We find an upper limit of about 20 pc for the outer scale of the magnetic interstellar turbulence toward the Fan region. We also find a variation of the ratio of random to ordered field as a function of Galactic coordinates, supporting different turbulent regimes. We use power spectra fluctuations from LOFAR as well as earlier GMRT and WSRT observations to constrain the outer scale of turbulence of the Galactic synchrotron foreground, finding a range of plausible values of 10-20 pc. Then, we use this information to deduce lower limits of the ratio of ordered to random magnetic field strength. These are found to be 0.3, 0.3, and 0.5 for the LOFAR, WSRT and GMRT fields considered respectively. Both these constraints are in agreement with previous estimates.

One of the most promising ways to study the epoch of reionization (EoR) is through radio observations of the redshifted 21-cm line emission from neutral hydrogen. These observations are complicated by the fact that the mapping of redshifts to line-of-sight positions is distorted by the peculiar velocities of the gas. Such distortions can be a source of error if they are not properly understood, but they also encode information about cosmology and astrophysics. We study the effects of redshift space distortions on the power spectrum of 21-cm radiation from the EoR using large scale $N$-body and radiative transfer simulations. We quantify the anisotropy introduced in the 21-cm power spectrum by redshift space distortions and show how it evolves as reionization progresses and how it relates to the underlying physics. We go on to study the effects of redshift space distortions on LOFAR observations, taking instrument noise and foreground subtraction into account. We find that LOFAR should be able to directly observe the power spectrum anisotropy due to redshift space distortions at spatial scales around $k \sim 0.1$ Mpc$^{-1}$ after $\gtrsim$ 1000 hours of integration time. At larger scales, sample errors become a limiting factor, while at smaller scales detector noise and foregrounds make the extraction of the signal problematic. Finally, we show how the astrophysical information contained in the evolution of the anisotropy of the 21-cm power spectrum can be extracted from LOFAR observations, and how it can be used to distinguish between different reionization scenarios.

A major stage of radio interferometric data processing is calibration or the estimation of systematic errors in the data and the correction for such errors. A stochastic error (noise) model is assumed, and in most cases, this underlying model is assumed to be Gaussian. However, outliers in the data due to interference or due to errors in the sky model would have adverse effects on processing based on a Gaussian noise model. Most of the shortcomings of calibration such as the loss in flux or coherence, and the appearance of spurious sources, could be attributed to the deviations of the underlying noise model. In this paper, we propose to improve the robustness of calibration by using a noise model based on Student's t distribution. Student's t noise is a special case of Gaussian noise when the variance is unknown. Unlike Gaussian noise model based calibration, traditional least squares minimization would not directly extend to a case when we have a Student's t noise model. Therefore, we use a variant of the Expectation Maximization (EM) algorithm, called the Expectation-Conditional Maximization Either (ECME) algorithm when we have a Student's t noise model and use the Levenberg-Marquardt algorithm in the maximization step. We give simulation results to show the robustness of the proposed calibration method as opposed to traditional Gaussian noise model based calibration, especially in preserving the flux of weaker sources that are not included in the calibration model.

The main objective of this work is to accelerate the Maximum-Likelihood (ML) estimation procedure in radio interferometric calibration. We introduce the OS-LS and the OS-SAGE radio interferometric calibration methods, as a combination of the Ordered-Subsets (OS) method with the Least-Squares (LS) and Space Alternating Generalized Expectation maximization (SAGE) calibration techniques, respectively. The OS algorithm speeds up the ML estimation and achieves nearly the same level of accuracy of solutions as the one obtained by the non-OS methods. We apply the OS-LS and OS-SAGE calibration methods to simulated observations and show that these methods have a much higher convergence rate relative to the conventional LS and SAGE techniques. Moreover, the obtained results show that the OS-SAGE calibration technique has a superior performance compared to the OS-LS calibration method in the sense of achieving more accurate results while having significantly less computational cost.

LOFAR, the LOw-Frequency ARray, is a new-generation radio interferometer constructed in the north of the Netherlands and across europe. Utilizing a novel phased-array design, LOFAR covers the largely unexplored low-frequency range from 10-240 MHz and provides a number of unique observing capabilities. Spreading out from a core located near the village of Exloo in the northeast of the Netherlands, a total of 40 LOFAR stations are nearing completion. A further five stations have been deployed throughout Germany, and one station has been built in each of France, Sweden, and the UK. Digital beam-forming techniques make the LOFAR system agile and allow for rapid repointing of the telescope as well as the potential for multiple simultaneous observations. With its dense core array and long interferometric baselines, LOFAR achieves unparalleled sensitivity and angular resolution in the low-frequency radio regime. The LOFAR facilities are jointly operated by the International LOFAR Telescope (ILT) foundation, as an observatory open to the global astronomical community. LOFAR is one of the first radio observatories to feature automated processing pipelines to deliver fully calibrated science products to its user community. LOFAR's new capabilities, techniques and modus operandi make it an important pathfinder for the Square Kilometre Array (SKA). We give an overview of the LOFAR instrument, its major hardware and software components, and the core science objectives that have driven its design. In addition, we present a selection of new results from the commissioning phase of this new radio observatory.

Using a combination of N-body simulations, semi-analytic models and radiative transfer calculations, we have estimated the theoretical cross power spectrum between galaxies and the 21cm emission from neutral hydrogen during the epoch of reionization. In accordance with previous studies, we find that the 21cm emission is initially correlated with halos on large scales (> 30 Mpc), anti-correlated on intermediate (~ 5 Mpc), and uncorrelated on small (< 3 Mpc) scales. This picture quickly changes as reionization proceeds and the two fields become anti-correlated on large scales. The normalization of the cross power spectrum can be used to set constraints on the average neutral fraction in the intergalactic medium and its shape can be a tool to study the topology of reionization. When we apply a drop-out technique to select galaxies and add to the 21cm signal the noise expected from the LOFAR telescope, we find that while the normalization of the cross power spectrum remains a useful tool for probing reionization, its shape becomes too noisy to be informative. On the other hand, for a Lyalpha Emitter (LAE) survey both the normalization and the shape of the cross power spectrum are suitable probes of reionization. A closer look at a specific planned LAE observing program using Subaru Hyper-Suprime Cam reveals concerns about the strength of the 21cm signal at the planned redshifts. If the ionized fraction at z ~ 7 is lower that the one estimated here, then using the cross power spectrum may be a useful exercise given that at higher redshifts and neutral fractions it is able to distinguish between two toy models with different topologies.

Faraday rotation measurements using the current and next generation of low-frequency radio telescopes will provide a powerful probe of astronomical magnetic fields. However, achieving the full potential of these measurements requires accurate removal of the time-variable ionospheric Faraday rotation contribution. We present ionFR, a code that calculates the amount of ionospheric Faraday rotation for a specific epoch, geographic location, and line-of-sight. ionFR uses a number of publicly available, GPS-derived total electron content maps and the most recent release of the International Geomagnetic Reference Field. We describe applications of this code for the calibration of radio polarimetric observations, and demonstrate the high accuracy of its modeled ionospheric Faraday rotations using LOFAR pulsar observations. These show that we can accurately determine some of the highest-precision pulsar rotation measures ever achieved. Precision rotation measures can be used to monitor rotation measure variations - either intrinsic or due to the changing line-of-sight through the interstellar medium. This calibration is particularly important for nearby sources, where the ionosphere can contribute a significant fraction of the observed rotation measure. We also discuss planned improvements to ionFR, as well as the importance of ionospheric Faraday rotation calibration for the emerging generation of low-frequency radio telescopes, such as the SKA and its pathfinders.

The Square Kilometre Array (SKA) will have a low frequency component (SKA-low) which has as one of its main science goals the study of the redshifted 21cm line from the earliest phases of star and galaxy formation in the Universe. This 21cm signal provides a new and unique window on both the formation of the first stars and accreting black holes and the later period of substantial ionization of the intergalactic medium. The signal will teach us fundamental new things about the earliest phases of structure formation, cosmology and even has the potential to lead to the discovery of new physical phenomena. Here we present a white paper with an overview of the science questions that SKA-low can address, how we plan to tackle these questions and what this implies for the basic design of the telescope.

In order to cope with the increased data volumes generated by modern radio interferometers such as LOFAR (Low Frequency Array) or SKA (Square Kilometre Array), fast and efficient calibration algorithms are essential. Traditional radio interferometric calibration is performed using nonlinear optimization techniques such as the Levenberg-Marquardt algorithm in Euclidean space. In this paper, we reformulate radio interferometric calibration as a nonlinear optimization problem on a Riemannian manifold. The reformulated calibration problem is solved using the Riemannian trust-region method. We show that calibration on a Riemannian manifold has faster convergence with reduced computational cost compared to conventional calibration in Euclidean space.

One of the key science projects of the Low-Frequency Array (LOFAR) is the detection of the cosmological signal coming from the Epoch of Reionization (EoR). Here we present the LOFAR EoR Diagnostic Database (LEDDB) that is used in the storage, management, processing and analysis of the LOFAR EoR observations. It stores referencing information of the observations and diagnostic parameters extracted from their calibration. This stored data is used to ease the pipeline processing, monitor the performance of the telescope and visualize the diagnostic parameters which facilitates the analysis of the several contamination effects on the signals. It is implemented with PostgreSQL and accessed through the psycopg2 python module. We have developed a very flexible query engine, which is used by a web user interface to access the database, and a very extensive set of tools for the visualization of the diagnostic parameters through all their multiple dimensions.

Cassiopeia A was observed using the Low-Band Antennas of the LOw Frequency ARray (LOFAR) with high spectral resolution. This allowed a search for radio recombination lines (RRLs) along the line-of-sight to this source. Five carbon-alpha RRLs were detected in absorption between 40 and 50 MHz with a signal-to-noise ratio of > 5 from two independent LOFAR datasets. The derived line velocities (v_LSR ~ -50 km/s) and integrated optical depths (~ 13 s^-1) of the RRLs in our spectra, extracted over the whole supernova remnant, are consistent within each LOFAR dataset and with those previously reported. For the first time, we are able to extract spectra against the brightest hotspot of the remnant at frequencies below 330 MHz. These spectra show significantly higher (15-80 %) integrated optical depths, indicating that there is small-scale angular structure on the order of ~1 pc in the absorbing gas distribution over the face of the remnant. We also place an upper limit of 3 x 10^-4 on the peak optical depths of hydrogen and helium RRLs. These results demonstrate that LOFAR has the desired spectral stability and sensitivity to study faint recombination lines in the decameter band.

Some radio pulsars show clear drifting subpulses, in which subpulses are seen to drift in pulse longitude in a systematic pattern. Here we examine how the drifting subpulses of PSR B0809+74 evolve with time and observing frequency. We show that the subpulse period (P3) is constant on timescales of days, months and years, and between 14-5100 MHz. Despite this, the shapes of the driftbands change radically with frequency. Previous studies have concluded that, while the subpulses appear to move through the pulse window approximately linearly at low frequencies (< 500 MHz), a discrete step of 180 degrees in subpulse phase is observed at higher frequencies (> 820 MHz) near to the peak of the average pulse profile. We use LOFAR, GMRT, GBT, WSRT and Effelsberg 100-m data to explore the frequency-dependence of this phase step. We show that the size of the subpulse phase step increases gradually, and is observable even at low frequencies. We attribute the subpulse phase step to the presence of two separate driftbands, whose relative arrival times vary with frequency - one driftband arriving 30 pulses earlier at 20 MHz than it does at 1380 MHz, whilst the other arrives simultaneously at all frequencies. The drifting pattern which is observed here cannot be explained by either the rotating carousel model or the surface oscillation model, and could provide new insight into the physical processes happening within the pulsar magnetosphere.

The aim of the LOFAR Epoch of Reionization (EoR) project is to detect the spectral fluctuations of the redshifted HI 21cm signal. This signal is weaker by several orders of magnitude than the astrophysical foreground signals and hence, in order to achieve this, very long integrations, accurate calibration for stations and ionosphere and reliable foreground removal are essential. One of the prospective observing windows for the LOFAR EoR project will be centered at the North Celestial Pole (NCP). We present results from observations of the NCP window using the LOFAR highband antenna (HBA) array in the frequency range 115 MHz to 163 MHz. The data were obtained in April 2011 during the commissioning phase of LOFAR. We used baselines up to about 30 km. With about 3 nights, of 6 hours each, effective integration we have achieved a noise level of about 100 microJy/PSF in the NCP window. Close to the NCP, the noise level increases to about 180 microJy/PSF, mainly due to additional contamination from unsubtracted nearby sources. We estimate that in our best night, we have reached a noise level only a factor of 1.4 above the thermal limit set by the noise from our Galaxy and the receivers. Our continuum images are several times deeper than have been achieved previously using the WSRT and GMRT arrays. We derive an analytical explanation for the excess noise that we believe to be mainly due to sources at large angular separation from the NCP.

The new generation of radio synthesis arrays, such as LOFAR and SKA, have been designed to surpass existing arrays in terms of sensitivity, angular resolution and frequency coverage. This evolution has led to the development of advanced calibration techniques that ensure the delivery of accurate results at the lowest possible computational cost. However, the performance of such calibration techniques is still limited by the compact, bright sources in the sky, used as calibrators. It is important to have a bright enough source that is well distinguished from the background noise level in order to achieve satisfactory results in calibration. We present "clustered calibration" as a modification to traditional radio interferometric calibration, in order to accommodate faint sources that are almost below the background noise level into the calibration process. The main idea is to employ the information of the bright sources' measured signals as an aid to calibrate fainter sources that are nearby the bright sources. In the case where we do not have bright enough sources, a source cluster could act as a bright source that can be distinguished from background noise. We construct a number of source clusters assuming that the signals of the sources belonging to a single cluster are corrupted by almost the same errors, and each cluster is calibrated as a single source, using the combined coherencies of its sources simultaneously. This upgrades the power of an individual faint source by the effective power of its cluster. We give performance analysis of clustered calibration to show the superiority of this approach compared to the traditional unclustered calibration. We also provide analytical criteria to choose the optimum number of clusters for a given observation in an efficient manner.

We discuss the feasibility of the detection of the 21cm forest in the diffuse IGM with the radio telescope LOFAR. The optical depth to the 21cm line has been derived using simulations of reionization which include detailed radiative transfer of ionizing photons. We find that the spectra from reionization models with similar total comoving hydrogen ionizing emissivity but different frequency distribution look remarkably similar. Thus, unless the reionization histories are very different from each other (e.g. a predominance of UV vs. x-ray heating) we do not expect to distinguish them by means of observations of the 21cm forest. Because the presence of a strong x-ray background would make the detection of 21cm line absorption impossible, the lack of absorption could be used as a probe of the presence/intensity of the x-ray background and the thermal history of the universe. Along a random line of sight LOFAR could detect a global suppression of the spectrum from z>12, when the IGM is still mostly neutral and cold, in contrast with the more well-defined, albeit broad, absorption features visible at lower redshift. Sharp, strong absorption features associated with rare, high density pockets of gas could be detected also at z~7 along preferential lines of sight.

Aims: This paper discusses the spectral occupancy for performing radio astronomy with the Low-Frequency Array (LOFAR), with a focus on imaging observations. Methods: We have analysed the radio-frequency interference (RFI) situation in two 24-h surveys with Dutch LOFAR stations, covering 30-78 MHz with low-band antennas and 115-163 MHz with high-band antennas. This is a subset of the full frequency range of LOFAR. The surveys have been observed with a 0.76 kHz / 1 s resolution. Results: We measured the RFI occupancy in the low and high frequency sets to be 1.8% and 3.2% respectively. These values are found to be representative values for the LOFAR radio environment. Between day and night, there is no significant difference in the radio environment. We find that lowering the current observational time and frequency resolutions of LOFAR results in a slight loss of flagging accuracy. At LOFAR's nominal resolution of 0.76 kHz and 1 s, the false-positives rate is about 0.5%. This rate increases approximately linearly when decreasing the data frequency resolution. Conclusions: Currently, by using an automated RFI detection strategy, the LOFAR radio environment poses no perceivable problems for sensitive observing. It remains to be seen if this is still true for very deep observations that integrate over tens of nights, but the situation looks promising. Reasons for the low impact of RFI are the high spectral and time resolution of LOFAR; accurate detection methods; strong filters and high receiver linearity; and the proximity of the antennas to the ground. We discuss some strategies that can be used once low-level RFI starts to become apparent. It is important that the frequency range of LOFAR remains free of broadband interference, such as DAB stations and windmills.

Full polarimetric radio interferometric calibration is performed by estimating 2 by 2 Jones matrices representing instrumental and propagation effects. The solutions obtained in this way differ from the true solutions by a 2 by 2 unitary matrix ambiguity. This ambiguity is common to all stations for which a solution is obtained but it is different for solutions obtained at different time and frequency intervals. Therefore, straightforward interpolation of solutions obtained at different time and frequency intervals is not possible. In this paper, we propose to use the theory of quotient manifolds for obtaining correct interpolants that are immune to unitary matrix ambiguities.

The knowledge of receiver beam shapes is essential for accurate radio interferometric imaging. Traditionally, this information is obtained by holographic techniques or by numerical simulation. However, such methods are not feasible for an observation with time varying beams, such as the beams produced by a phased array radio interferometer. We propose the use of the observed data itself for the estimation of the beam shapes. We use the directional gains obtained along multiple sources across the sky for the construction of a time varying beam model. The construction of this model is an ill posed non linear optimization problem. Therefore, we propose to use Riemannian optimization, where we consider the constraints imposed as a manifold. We compare the performance of the proposed approach with traditional unconstrained optimization and give results to show the superiority of the proposed approach.

The first generation of redshifted 21 cm detection experiments, carried out with arrays like LOFAR, MWA and GMRT, will have a very low signal-to-noise ratio per resolution element (\sim 0.2). In addition, whereas the variance of the cosmological signal decreases on scales larger than the typical size of ionization bubbles, the variance of the formidable galactic foregrounds increases, making it hard to disentangle the two on such large scales. The poor sensitivity on small scales on the one hand, and the foregrounds effect on large scales on the other hand, make direct imaging of the Epoch of Reionization of the Universe very difficult, and detection of the signal therefore is expected to be statistical.Despite these hurdles, in this paper we argue that for many reionization scenarios low resolution images could be obtained from the expected data. This is because at the later stages of the process one still finds very large pockets of neutral regions in the IGM, reflecting the clustering of the large-scale structure, which stays strong up to scales of \sim 120 comoving Mpc/h (\sim 1 degree). The coherence of the emission on those scales allows us to reach sufficient S/N (\sim 3) so as to obtain reionization 21 cm images. Such images will be extremely valuable for answering many cosmological questions but above all they will be a very powerful tool to test our control of the systematics in the data. The existence of this typical scale (\sim 120 comoving Mpc/h) also argues for designing future EoR experiments, e.g., with SKA, with a field of view of at least 4 degree.

Dispersion in the interstellar medium is a well known phenomenon that follows a simple relationship, which has been used to predict the time delay of dispersed radio pulses since the late 1960s. We performed wide-band simultaneous observations of four pulsars with LOFAR (at 40-190 MHz), the 76-m Lovell Telescope (at 1400 MHz) and the Effelsberg 100-m Telescope (at 8000 MHz) to test the accuracy of the dispersion law over a broad frequency range. In this paper we present the results of these observations which show that the dispersion law is accurate to better than 1 part in 100000 across our observing band. We use this fact to constrain some of the properties of the ISM along the line-of-sight and use the lack of any aberration or retardation effects to determine upper limits on emission heights in the pulsar magnetosphere. We also discuss the effect of pulse profile evolution on our observations, and the implications that it could have for precision pulsar timing projects such as the detection of gravitational waves with pulsar timing arrays.

Interferometric calibration always yields non unique solutions. It is therefore essential to remove these ambiguities before the solutions could be used in any further modeling of the sky, the instrument or propagation effects such as the ionosphere. We present a method for LOFAR calibration which does not yield a unitary ambiguity, especially under ionospheric distortions. We also present exact ambiguities we get in our solutions, in closed form. Casting this as an optimization problem, we also present conditions for this approach to work. The proposed method enables us to use the solutions obtained via calibration for further modeling of instrumental and propagation effects. We provide extensive simulation results on the performance of our method. Moreover, we also give cases where due to degeneracy, this method fails to perform as expected and in such cases, we suggest exploiting diversity in time, space and frequency.

The application of orthonormal basis functions such as Prolate Spheroidal Wave Functions (PSWF) for accurate source modeling in radio astronomy has been comprehensively studied. They are of great importance for high fidelity, high dynamic range imaging with new radio telescopes as well as conventional ones. But the construction of PSWF is computationally expensive compared to other closed form basis functions. In this paper, we suggest a solution to reduce its computational cost by more efficient construction of the matrix kernel which relates the image domain to visibility (or Fourier) domain. Radio astronomical images are mostly represented using a regular grid of rectangular pixels. This is required for efficient storage and display purposes and moreover, comes naturally as a by product of the Fast Fourier Transform (FFT) in imaging. We propose the use of Delaunay triangulation as opposed to regular gridding of an image for a finer selection of the region of interest (signal support) during the PSWF kernel construction. We show that the computational efficiency improves without loss of information. Once the PSWF basis is constructed using the irregular grid, we revert back to the regular grid by interpolation and thereafter, conventional imaging techniques can be applied.

This paper introduces an amendment to radio interferometric calibration of sources below the noise level. The main idea is to employ the information of the stronger sources' measured signals as a plug-in criterion to solve for the weaker ones. For this purpose, we construct a number of source clusters, with centroids mainly near the strongest sources, assuming that the signals of the sources belonging to a single cluster are corrupted by almost the same errors. Due to this characteristic of clusters, each cluster is calibrated as a single source, using all the coherencies of its sources simultaneously. The obtained solutions for every cluster are assigned to all the cluster's sources. An illustrative example reveals the superiority of this calibration compared to the un-clustered calibration.

Low frequency radio waves, while challenging to observe, are a rich source of information about pulsars. The LOw Frequency ARray (LOFAR) is a new radio interferometer operating in the lowest 4 octaves of the ionospheric "radio window": 10-240MHz, that will greatly facilitate observing pulsars at low radio frequencies. Through the huge collecting area, long baselines, and flexible digital hardware, it is expected that LOFAR will revolutionize radio astronomy at the lowest frequencies visible from Earth. LOFAR is a next-generation radio telescope and a pathfinder to the Square Kilometre Array (SKA), in that it incorporates advanced multi-beaming techniques between thousands of individual elements. We discuss the motivation for low-frequency pulsar observations in general and the potential of LOFAR in addressing these science goals. We present LOFAR as it is designed to perform high-time-resolution observations of pulsars and other fast transients, and outline the various relevant observing modes and data reduction pipelines that are already or will soon be implemented to facilitate these observations. A number of results obtained from commissioning observations are presented to demonstrate the exciting potential of the telescope. This paper outlines the case for low frequency pulsar observations and is also intended to serve as a reference for upcoming pulsar/fast transient science papers with LOFAR.

The aim of the new generation of radio synthesis arrays such as LOFAR and SKA is to achieve much higher sensitivity, resolution and frequency coverage than what is available now, especially at low frequencies. To accomplish this goal, the accuracy of the calibration techniques used is of considerable importance. Moreover, since these telescopes produce huge amounts of data, speed of convergence of calibration is a major bottleneck. The errors in calibration are due to system noise (sky and instrumental) as well as the estimation errors introduced by the calibration technique itself, which we call solver noise. We define solver noise as the distance between the optimal solution (the true value of the unknowns, uncorrupted by the system noise) and the solution obtained by calibration. We present the Space Alternating Generalized Expectation Maximization (SAGE) calibration technique, which is a modification of the Expectation Maximization algorithm, and compare its performance with the traditional Least Squares calibration based on the level of solver noise introduced by each technique. For this purpose, we develop statistical methods that use the calibrated solutions to estimate the level of solver noise. The SAGE calibration algorithm yields very promising results both in terms of accuracy and speed of convergence. The comparison approaches we adopt introduce a new framework for assessing the performance of different calibration schemes.

In order to produce high dynamic range images in radio interferometry, bright extended sources need to be removed with minimal error. However, this is not a trivial task because the Fourier plane is sampled only at a finite number of points. The ensuing deconvolution problem has been solved in many ways, mainly by algorithms based on CLEAN. However, such algorithms that use image pixels as basis functions have inherent limitations and by using an orthonormal basis that span the whole image, we can overcome them. The construction of such an orthonormal basis involves fine tuning of many free parameters that define the basis functions. The optimal basis for a given problem (or a given extended source) is not guaranteed. In this paper, we discuss the use of generalized prolate spheroidal wave functions as a basis. Given the geometry (or the region of interest) of an extended source and the sampling points on the visibility plane, we can construct the optimal basis to model the source. Not only does this gives us the minimum number of basis functions required but also the artifacts outside the region of interest are minimized.

In the coming years a new insight into galaxy formation and the thermal history of the Universe is expected to come from the detection of the highly redshifted cosmological 21 cm line. The cosmological 21 cm line signal is buried under Galactic and extragalactic foregrounds which are likely to be a few orders of magnitude brighter. Strategies and techniques for effective subtraction of these foreground sources require a detailed knowledge of their structure in both intensity and polarization on the relevant angular scales of 1-30 arcmin. We present results from observations conducted with the Westerbork telescope in the 140-160 MHz range with 2 arcmin resolution in two fields located at intermediate Galactic latitude, centred around the bright quasar 3C196 and the North Celestial Pole. They were observed with the purpose of characterizing the foreground properties in sky areas where actual observations of the cosmological 21 cm line could be carried out. The polarization data were analysed through the rotation measure synthesis technique. We have computed total intensity and polarization angular power spectra. Total intensity maps were carefully calibrated, reaching a high dynamic range, 150000:1 in the case of the 3C196 field. [abridged]

Deconvolution is essential for radio interferometric imaging to produce scientific quality data because of finite sampling in the Fourier plane. Most deconvolution algorithms are based on CLEAN which uses a grid of image pixels, or clean components. A critical matter in this process is the selection of pixel size for optimal results in deconvolution. As a rule of thumb, the pixel size is chosen smaller than the resolution dictated by the interferometer. For images consisting of unresolved (or point like) sources, this approach yields optimal results. However, for sources that are not point like, in particular for partially resolved sources, the selection of right pixel size is still an open issue. In this paper, we investigate the limitations of pixelization in deconvolving extended sources. In particular, we pursue the usage of orthonormal basis functions to model extended sources yielding better results than by using clean components.

One of the aims of the Low Frequency Array (LOFAR) Epoch of Reionization (EoR) project is to measure the power spectrum of variations in the intensity of redshifted 21-cm radiation from the EoR. The sensitivity with which this power spectrum can be estimated depends on the level of thermal noise and sample variance, and also on the systematic errors arising from the extraction process, in particular from the subtraction of foreground contamination. We model the extraction process using realistic simulations of the cosmological signal, the foregrounds and noise, and so estimate the sensitivity of the LOFAR EoR experiment to the redshifted 21-cm power spectrum. Detection of emission from the EoR should be possible within 360 hours of observation with a single station beam. Integrating for longer, and synthesizing multiple station beams within the primary (tile) beam, then enables us to extract progressively more accurate estimates of the power at a greater range of scales and redshifts. We discuss different observational strategies which compromise between depth of observation, sky coverage and frequency coverage. A plan in which lower frequencies receive a larger fraction of the time appears to be promising. We also study the nature of the bias which foreground fitting errors induce on the inferred power spectrum, and discuss how to reduce and correct for this bias. The angular and line-of-sight power spectra have different merits in this respect, and we suggest considering them separately in the analysis of LOFAR data.

We present the first results from a series of observations conducted with the Westerbork telescope in the 140--160 MHz range with a 2 arcmin resolution aimed at characterizing the properties of the foregrounds for epoch of reionization experiments. For the first time we have detected fluctuations in the Galactic diffuse emission on scales greater than 13 arcmin at 150 MHz, in the low Galactic latitude area known as Fan region. Those fluctuations have an $rms$ of 14 K. The total intensity power spectrum shows a power--law behaviour down to $\ell \sim 900$ with slope $\beta^I_\ell = -2.2 \pm 0.3$. The detection of diffuse emission at smaller angular scales is limited by residual point sources. We measured an $rms$ confusion noise of $\sim$3 mJy beam$^{-1}$. Diffuse polarized emission was also detected for the first time at this frequency. The polarized signal shows complex structure both spatially and along the line of sight. The polarization power spectrum shows a power--law behaviour down to $\ell \sim 2700$ with slope $\beta^P_\ell = -1.65 \pm 0.15$. The $rms$ of polarization fluctuations is 7.2 K on 4 arcmin scales. By extrapolating the measured spectrum of total intensity emission, we find a contamination on the cosmological signal of $\delta T= \sqrt{\ell (\ell+1) C^I_\ell / 2\pi} \sim 5.7$ K on 5 arcmin scales and a corresponding $rms$ value of $\sim$18.3 K at the same angular scale. The level of the polarization power spectrum is $\delta T \sim 3.3$ K on 5 arcmin scales. Given its exceptionally bright polarized signal, the Fan region is likely to represent an upper limit on the sky brightness at moderate and high Galactic latitude.

An obstacle to the detection of redshifted 21cm emission from the epoch of reionization (EoR) is the presence of foregrounds which exceed the cosmological signal in intensity by orders of magnitude. We argue that in principle it would be better to fit the foregrounds non-parametrically - allowing the data to determine their shape - rather than selecting some functional form in advance and then fitting its parameters. Non-parametric fits often suffer from other problems, however. We discuss these before suggesting a non-parametric method, Wp smoothing, which seems to avoid some of them. After outlining the principles of Wp smoothing we describe an algorithm used to implement it. We then apply Wp smoothing to a synthetic data cube for the LOFAR EoR experiment. The performance of Wp smoothing, measured by the extent to which it is able to recover the variance of the cosmological signal and to which it avoids leakage of power from the foregrounds, is compared to that of a parametric fit, and to another non-parametric method (smoothing splines). We find that Wp smoothing is superior to smoothing splines for our application, and is competitive with parametric methods even though in the latter case we may choose the functional form of the fit with advance knowledge of the simulated foregrounds. Finally, we discuss how the quality of the fit is affected by the frequency resolution and range, by the characteristics of the cosmological signal and by edge effects.

Detecting redshifted 21cm emission from neutral hydrogen in the early Universe promises to give direct constraints on the epoch of reionization (EoR). It will, though, be very challenging to extract the cosmological signal (CS) from foregrounds and noise which are orders of magnitude larger. Fortunately, the signal has some characteristics which differentiate it from the foregrounds and noise, and we suggest that using the correct statistics may tease out signatures of reionization. We generate mock datacubes simulating the output of the Low Frequency Array (LOFAR) EoR experiment. These cubes combine realistic models for Galactic and extragalactic foregrounds and the noise with three different simulations of the CS. We fit out the foregrounds, which are smooth in the frequency direction, to produce residual images in each frequency band. We denoise these images and study the skewness of the one-point distribution in the images as a function of frequency. We find that, under sufficiently optimistic assumptions, we can recover the main features of the redshift evolution of the skewness in the 21cm signal. We argue that some of these features - such as a dip at the onset of reionization, followed by a rise towards its later stages - may be generic, and give us a promising route to a statistical detection of reionization.

A number of experiments are set to measure the 21-cm signal of neutral hydrogen from the Epoch of Reionization (EoR). The common denominator of these experiments are the large data sets produced, contaminated by various instrumental effects, ionospheric distortions, RFI and strong Galactic and extragalactic foregrounds. In this paper, the first in a series, we present the Data Model that will be the basis of the signal analysis for the LOFAR (Low Frequency Array) EoR Key Science Project (LOFAR EoR KSP). Using this data model we simulate realistic visibility data sets over a wide frequency band, taking properly into account all currently known instrumental corruptions (e.g. direction-dependent gains, complex gains, polarization effects, noise, etc). We then apply primary calibration errors to the data in a statistical sense, assuming that the calibration errors are random Gaussian variates at a level consistent with our current knowledge based on observations with the LOFAR Core Station 1. Our aim is to demonstrate how the systematics of an interferometric measurement affect the quality of the calibrated data, how errors correlate and propagate, and in the long run how this can lead to new calibration strategies. We present results of these simulations and the inversion process and extraction procedure. We also discuss some general properties of the coherency matrix and Jones formalism that might prove useful in solving the calibration problem of aperture synthesis arrays. We conclude that even in the presence of realistic noise and instrumental errors, the statistical signature of the EoR signal can be detected by LOFAR with relatively small errors. A detailed study of the statistical properties of our data model and more complex instrumental models will be considered in the future.

Radio Interferometry is an essential method for astronomical observations. Self-calibration techniques have increased the quality of the radio astronomical observations (and hence the science) by orders of magnitude. Recently, there is a drive towards sensor arrays built using inexpensive hardware and distributed over a wide area acting as radio interferometers. Calibration of such arrays poses new problems in terms of computational cost as well as in performance of existing calibration algorithms. We consider the application of the Space Alternating Generalized Expectation Maximization (SAGE) \cite{Fess94} algorithm for calibration of radio interferometric arrays. Application to real data shows that this is an improvement over existing calibration algorithms that are based on direct, deterministic non linear optimization. As presented in this paper, we can improve the computational cost as well as the quality of the calibration using this algorithm.

We present an efficient method to generate large simulations of the Epoch of Reionization (EoR) without the need for a full 3-dimensional radiative transfer code. Large dark-matter-only simulations are post-processed to produce maps of the redshifted 21cm emission from neutral hydrogen. Dark matter haloes are embedded with sources of radiation whose properties are either based on semi-analytical prescriptions or derived from hydrodynamical simulations. These sources could either be stars or power-law sources with varying spectral indices. Assuming spherical symmetry, ionized bubbles are created around these sources, whose radial ionized fraction and temperature profiles are derived from a catalogue of 1-D radiative transfer experiments. In case of overlap of these spheres, photons are conserved by redistributing them around the connected ionized regions corresponding to the spheres. The efficiency with which these maps are created allows us to span the large parameter space typically encountered in reionization simulations. We compare our results with other, more accurate, 3-D radiative transfer simulations and find excellent agreement for the redshifts and the spatial scales of interest to upcoming 21cm experiments. We generate a contiguous observational cube spanning redshift 6 to 12 and use these simulations to study the differences in the reionization histories between stars and quasars. Finally, the signal is convolved with the LOFAR beam response and its effects are analyzed and quantified. Statistics performed on this mock data set shed light on possible observational strategies for LOFAR.

Radio astronomical observations have very poor signal to noise ratios, unlike in other disciplines. On the other hand, it is possible to observe the object of interest for long time intervals as well as using a wider bandwidth. Traditionally, by averaging in time and in frequency, it has been possible to improve the signal to noise ratio of astronomical observations to improve the dynamic range. This is possible due to the inherent assumption that the object of interest in the sky is invariant over time and the frequency range of observation. However, in reality this assumption does not hold, due to intrinsic variation of the sky as well as due to errors generated by the instrument. In this paper, we shall discuss an alternative to averaging of images, without ignoring subtle changes in the observed data over time and frequency, using subspace decomposition. By separation of data to signal and noise subspaces, not only would this improve the quality of the data, but also enable us to detect faint artifacts due to calibration errors, interference etc.

Future high redshift 21-cm experiments will suffer from a high degree of contamination, due both to astrophysical foregrounds and to non-astrophysical and instrumental effects. In order to reliably extract the cosmological signal from the observed data, it is essential to understand very well all data components and their influence on the extracted signal. Here we present simulated astrophysical foregrounds datacubes and discuss their possible statistical effects on the data. The foreground maps are produced assuming 5 deg x 5 deg windows that match those expected to be observed by the LOFAR Epoch-of-Reionization (EoR) key science project. We show that with the expected LOFAR-EoR sky and receiver noise levels, which amount to ~52 mK at 150 MHz after 300 hours of total observing time, a simple polynomial fit allows a statistical reconstruction of the signal. We also show that the polynomial fitting will work for maps with realistic yet idealised instrument response, i.e., a response that includes only a uniform uv coverage as a function of frequency and ignores many other uncertainties. Polarized galactic synchrotron maps that include internal polarization and a number of Faraday screens along the line of sight are also simulated. The importance of these stems from the fact that the LOFAR instrument, in common with all current interferometric EoR experiments has an instrumentally polarized response.