We propose a probabilistic framework for performing simultaneous estimation of source structure and fringe-fitting parameters in Very Long Baseline Interferometry (VLBI) observations. As a first step, we demonstrate this technique through the analysis of synthetic short-duration Event Horizon Telescope (EHT) observations of various geometric source models at 230 GHz, in the presence of baseline-dependent thermal noise. We perform Bayesian parameter estimation and model selection between the different source models to obtain reliable uncertainty estimates and correlations between various source and fringe-fitting related model parameters. We also compare the Bayesian posteriors with those obtained using widely-used VLBI data reduction packages such as CASA and AIPS, by fringe-fitting 200 Monte Carlo simulations of each source model with different noise realisations, to obtain distributions of the Maximum A Posteriori (MAP) estimates. We find that, in the presence of resolved asymmetric source structure and a given array geometry, the traditional practice of fringe-fitting with a point source model yields appreciable offsets in the estimated phase residuals, potentially biasing or limiting the dynamic range of the starting model used for self-calibration. Simultaneously estimating the source structure earlier in the calibration process with formal uncertainties improves the precision and accuracy of fringe-fitting and establishes the potential of the available data especially when there is little prior information. We also note the potential applications of this method to astrometry and geodesy for specific science cases and the planned improvements to the computational performance and analyses of more complex source distributions.

Realistic synthetic observations of theoretical source models are essential for our understanding of real observational data. In using synthetic data, one can verify the extent to which source parameters can be recovered and evaluate how various data corruption effects can be calibrated. These studies are important when proposing observations of new sources, in the characterization of the capabilities of new or upgraded instruments, and when verifying model-based theoretical predictions in a comparison with observational data. We present the SYnthetic Measurement creator for long Baseline Arrays (SYMBA), a novel synthetic data generation pipeline for Very Long Baseline Interferometry (VLBI) observations. SYMBA takes into account several realistic atmospheric, instrumental, and calibration effects. We used SYMBA to create synthetic observations for the Event Horizon Telescope (EHT), a mm VLBI array, which has recently captured the first image of a black hole shadow. After testing SYMBA with simple source and corruption models, we study the importance of including all corruption and calibration effects. Based on two example general relativistic magnetohydrodynamics (GRMHD) model images of M87, we performed case studies to assess the attainable image quality with the current and future EHT array for different weather conditions. The results show that the effects of atmospheric and instrumental corruptions on the measured visibilities are significant. Despite these effects, we demonstrate how the overall structure of the input models can be recovered robustly after performing calibration steps. With the planned addition of new stations to the EHT array, images could be reconstructed with higher angular resolution and dynamic range. In our case study, these improvements allowed for a distinction between a thermal and a non-thermal GRMHD model based on salient features in reconstructed images.

(Abridged) The CASA software suite, can now reduce very long baseline interferometry (VLBI) data with the recent addition of a fringe fitter. Here, we present the Radboud PIpeline for the Calibration of high Angular Resolution Data (rPICARD), which is an open-source VLBI calibration and imaging pipeline built on top of the CASA framework. The pipeline is capable of reducing data from different VLBI arrays. It can be run non-interactively after only a few non-default input parameters are set and delivers high-quality calibrated data. CPU scalability based on a message-passing interface (MPI) implementation ensures that large bandwidth data from future arrays can be processed within reasonable computing times. Phase calibration is done with a Schwab-Cotton fringe fit algorithm. For the calibration of residual atmospheric effects, optimal solution intervals are determined based on the signal-to-noise ratio (S/N) of the data for each scan. Different solution intervals can be set for different antennas in the same scan to increase the number of detections in the low S/N regime. These novel techniques allow rPICARD to calibrate data from different arrays, including high-frequency and low-sensitivity arrays. The amplitude calibration is based on standard telescope metadata, and a robust algorithm can solve for atmospheric opacity attenuation in the high-frequency regime. Standard CASA tasks are used for CLEAN imaging and self-calibration. In this work we demonstrate the capabilities of rPICARD by calibrating and imaging 7 mm VLBA data of the central radio source in the M87 galaxy. The reconstructed jet image reveals a complex collimation profile and edge-brightened structure. A potential counter-jet is detected that has 10 % of the brightness of the approaching jet. This constrains jet speeds close to the radio core to about half the speed of light for small inclination angles.

Currently, HOPS and AIPS are the primary choices for the time-consuming process of (millimeter) Very Long Baseline Interferometry (VLBI) data calibration. However, for a full end-to-end pipeline, they either lack the ability to perform easily scriptable incremental calibration or do not provide full control over the workflow with the ability to manipulate and edit calibration solutions directly. The Common Astronomy Software Application (CASA) offers all these abilities, together with a secure development future and an intuitive Python interface, which is very attractive for young radio astronomers. Inspired by the recent addition of a global fringe-fitter, the capability to convert FITS-IDI files to measurement sets, and amplitude calibration routines based on ANTAB metadata, we have developed the the CASA-based Radboud PIpeline for the Calibration of high Angular Resolution Data (rPICARD). The pipeline will be able to handle data from multiple arrays: EHT, GMVA, VLBA and the EVN in the first release. Polarization and phase-referencing calibration are supported and a spectral line mode will be added in the future. The large bandwidths of future radio observatories ask for a scalable reduction software. Within CASA, a message passing interface (MPI) implementation is used for parallelization, reducing the total time needed for processing. The most significant gain is obtained for the time-consuming fringe-fitting task where each scan be processed in parallel.

In recent years new functionality for VLBI data processing has been added to the CASA package. This paper presents the new CASA tasks 'fringefit' and 'accor', which are closely matched to their AIPS counterparts FRING and ACCOR. Several CASA tasks received upgrades to handle VLBI specific metadata. With the current CASA release VLBI data processing is possible, and functionality will be expanded in the upcoming release. Longer term developments include fringe fitting of broad, non-continuous frequency bands and dispersive delays, which will ensure that the number of use cases for VLBI calibration will increase in future CASA releases.

We present a technique to automatically minimise the re-computation when a data analysis program is iteratively changed, or added to, as is often the case in exploratory data analysis in astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe the implementation of this technique for the CASA radio astronomy data reduction package. We also propose a technique for optimising efficiency of storage of memoized intermediate data products using copy-on-write and block level de-duplication and measure their practical efficiency. We find the minimal recomputation technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result. It also aids exploratory data analysis on batch-schedule cluster computer systems.

We present a technique to automatically minimise the re-computation when a data processing program is iteratively changed, or added to, as is often the case in exploratory data analysis in radio astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe a prototype implementation for the CASA data reduction package. This technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result.

The closest ever fly-by of the Martian moon Phobos, performed by the European Space Agency's Mars Express spacecraft, gives a unique opportunity to sharpen and test the Planetary Radio Interferometry and Doppler Experiments (PRIDE) technique in the interest of studying planet - satellite systems. The aim of this work is to demonstrate a technique of providing high precision positional and Doppler measurements of planetary spacecraft using the Mars Express spacecraft. The technique will be used in the framework of Planetary Radio Interferometry and Doppler Experiments in various planetary missions, in particular in fly-by mode. We advanced a novel approach to spacecraft data processing using the techniques of Doppler and phase-referenced very long baseline interferometry spacecraft tracking. We achieved, on average, mHz precision (30 {\mu}m/s at a 10 seconds integration time) for radial three-way Doppler estimates and sub-nanoradian precision for lateral position measurements, which in a linear measure (at a distance of 1.4 AU) corresponds to ~50 m.

In this paper a description is given of the SFXC software correlator, developed and maintained at the Joint Institute for VLBI in Europe (JIVE). The software is designed to run on generic Linux-based computing clusters. The correlation algorithm is explained in detail, as are some of the novel modes that software correlation has enabled, such as wide-field VLBI imaging through the use of multiple phase centres and pulsar gating and binning. This is followed by an overview of the software architecture. Finally, the performance of the correlator as a function of number of CPU cores, telescopes and spectral channels is shown.

A 10-arcmin field around the HDF(N) contains 92 radio sources >40 uJy, resolved by MERLIN+VLA at 0".2-2".0 resolution. 55 have Chandra X-ray counterparts including 18 with a hard X-ray photon index and high luminosity characteristic of a type-II (obscured) AGN. >70% of the radio sources have been classified as starbursts or AGN using radio morphologies, spectral indices and comparisons with optical appearance and MIR emission. Starbursts outnumber radio AGN 3:1. This study extends the VO methods previously used to identify X-ray-selected obscured type-II AGN to investigate whether very luminous radio and X-ray emission originates from different phenomena in the same galaxy. The high-redshift starbursts have typical sizes of 5--10 kpc and star formation rates of ~1000 Msun/yr. There is no correlation between radio and X-ray luminosities nor spectral indices at z>~1.3. ~70% of both the radio-selected AGN and the starburst samples were detected by Chandra. The X-ray luminosity indicates the presence of an AGN in at least half of the 45 cross-matched radio starbursts, of which 11 are type-II AGN including 7 at z>1.5. This distribution overlaps closely with the X-ray detected radio sources which were also detected by SCUBA. Stacked 1.4-GHz emission at the positions of radio-faint X-ray sources is correlated with X-ray hardness. Most extended radio starbursts at z>1.3 host X-ray selected obscured AGN. Radio emission from most of these ultra-luminous objects is dominated by star formation but it contributes less than 1/3 of their X-ray luminosity. Our results support the inferences from SCUBA and IR data, that at z>1.5, star formation is an order of magnitude more extended and more copious, it is closely linked to AGN activity and it is triggered differently, compared with star formation at lower redshifts.

We use Virtual Observatory methods to investigate the association between radio and X-ray emission at high redshifts. Fifty-five of the 92 HDF(N) sources resolved by combining MERLIN+VLA data were detected by Chandra, of which 18 are hard enough and bright enough to be obscured AGN. The high-z population of microJy radio sources is dominated by starbursts an order of magnitude more active and more extended than any found at z<1 and at least a quarter of these simultaneously host highly X-ray-luminous obscured AGN.