We propose a probabilistic framework for performing simultaneous estimation of source structure and fringe-fitting parameters in Very Long Baseline Interferometry (VLBI) observations. As a first step, we demonstrate this technique through the analysis of synthetic short-duration Event Horizon Telescope (EHT) observations of various geometric source models at 230 GHz, in the presence of baseline-dependent thermal noise. We perform Bayesian parameter estimation and model selection between the different source models to obtain reliable uncertainty estimates and correlations between various source and fringe-fitting related model parameters. We also compare the Bayesian posteriors with those obtained using widely-used VLBI data reduction packages such as CASA and AIPS, by fringe-fitting 200 Monte Carlo simulations of each source model with different noise realisations, to obtain distributions of the Maximum A Posteriori (MAP) estimates. We find that, in the presence of resolved asymmetric source structure and a given array geometry, the traditional practice of fringe-fitting with a point source model yields appreciable offsets in the estimated phase residuals, potentially biasing or limiting the dynamic range of the starting model used for self-calibration. Simultaneously estimating the source structure earlier in the calibration process with formal uncertainties improves the precision and accuracy of fringe-fitting and establishes the potential of the available data especially when there is little prior information. We also note the potential applications of this method to astrometry and geodesy for specific science cases and the planned improvements to the computational performance and analyses of more complex source distributions.

Realistic synthetic observations of theoretical source models are essential for our understanding of real observational data. In using synthetic data, one can verify the extent to which source parameters can be recovered and evaluate how various data corruption effects can be calibrated. These studies are important when proposing observations of new sources, in the characterization of the capabilities of new or upgraded instruments, and when verifying model-based theoretical predictions in a comparison with observational data. We present the SYnthetic Measurement creator for long Baseline Arrays (SYMBA), a novel synthetic data generation pipeline for Very Long Baseline Interferometry (VLBI) observations. SYMBA takes into account several realistic atmospheric, instrumental, and calibration effects. We used SYMBA to create synthetic observations for the Event Horizon Telescope (EHT), a mm VLBI array, which has recently captured the first image of a black hole shadow. After testing SYMBA with simple source and corruption models, we study the importance of including all corruption and calibration effects. Based on two example general relativistic magnetohydrodynamics (GRMHD) model images of M87, we performed case studies to assess the attainable image quality with the current and future EHT array for different weather conditions. The results show that the effects of atmospheric and instrumental corruptions on the measured visibilities are significant. Despite these effects, we demonstrate how the overall structure of the input models can be recovered robustly after performing calibration steps. With the planned addition of new stations to the EHT array, images could be reconstructed with higher angular resolution and dynamic range. In our case study, these improvements allowed for a distinction between a thermal and a non-thermal GRMHD model based on salient features in reconstructed images.

(Abridged) The CASA software suite, can now reduce very long baseline interferometry (VLBI) data with the recent addition of a fringe fitter. Here, we present the Radboud PIpeline for the Calibration of high Angular Resolution Data (rPICARD), which is an open-source VLBI calibration and imaging pipeline built on top of the CASA framework. The pipeline is capable of reducing data from different VLBI arrays. It can be run non-interactively after only a few non-default input parameters are set and delivers high-quality calibrated data. CPU scalability based on a message-passing interface (MPI) implementation ensures that large bandwidth data from future arrays can be processed within reasonable computing times. Phase calibration is done with a Schwab-Cotton fringe fit algorithm. For the calibration of residual atmospheric effects, optimal solution intervals are determined based on the signal-to-noise ratio (S/N) of the data for each scan. Different solution intervals can be set for different antennas in the same scan to increase the number of detections in the low S/N regime. These novel techniques allow rPICARD to calibrate data from different arrays, including high-frequency and low-sensitivity arrays. The amplitude calibration is based on standard telescope metadata, and a robust algorithm can solve for atmospheric opacity attenuation in the high-frequency regime. Standard CASA tasks are used for CLEAN imaging and self-calibration. In this work we demonstrate the capabilities of rPICARD by calibrating and imaging 7 mm VLBA data of the central radio source in the M87 galaxy. The reconstructed jet image reveals a complex collimation profile and edge-brightened structure. A potential counter-jet is detected that has 10 % of the brightness of the approaching jet. This constrains jet speeds close to the radio core to about half the speed of light for small inclination angles.

Currently, HOPS and AIPS are the primary choices for the time-consuming process of (millimeter) Very Long Baseline Interferometry (VLBI) data calibration. However, for a full end-to-end pipeline, they either lack the ability to perform easily scriptable incremental calibration or do not provide full control over the workflow with the ability to manipulate and edit calibration solutions directly. The Common Astronomy Software Application (CASA) offers all these abilities, together with a secure development future and an intuitive Python interface, which is very attractive for young radio astronomers. Inspired by the recent addition of a global fringe-fitter, the capability to convert FITS-IDI files to measurement sets, and amplitude calibration routines based on ANTAB metadata, we have developed the the CASA-based Radboud PIpeline for the Calibration of high Angular Resolution Data (rPICARD). The pipeline will be able to handle data from multiple arrays: EHT, GMVA, VLBA and the EVN in the first release. Polarization and phase-referencing calibration are supported and a spectral line mode will be added in the future. The large bandwidths of future radio observatories ask for a scalable reduction software. Within CASA, a message passing interface (MPI) implementation is used for parallelization, reducing the total time needed for processing. The most significant gain is obtained for the time-consuming fringe-fitting task where each scan be processed in parallel.

In recent years new functionality for VLBI data processing has been added to the CASA package. This paper presents the new CASA tasks 'fringefit' and 'accor', which are closely matched to their AIPS counterparts FRING and ACCOR. Several CASA tasks received upgrades to handle VLBI specific metadata. With the current CASA release VLBI data processing is possible, and functionality will be expanded in the upcoming release. Longer term developments include fringe fitting of broad, non-continuous frequency bands and dispersive delays, which will ensure that the number of use cases for VLBI calibration will increase in future CASA releases.

We present a technique to automatically minimise the re-computation when a data analysis program is iteratively changed, or added to, as is often the case in exploratory data analysis in astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe the implementation of this technique for the CASA radio astronomy data reduction package. We also propose a technique for optimising efficiency of storage of memoized intermediate data products using copy-on-write and block level de-duplication and measure their practical efficiency. We find the minimal recomputation technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result. It also aids exploratory data analysis on batch-schedule cluster computer systems.

We present a technique to automatically minimise the re-computation when a data processing program is iteratively changed, or added to, as is often the case in exploratory data analysis in radio astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe a prototype implementation for the CASA data reduction package. This technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result.