Normalized to: Varbanescu, A.
[1]
oai:arXiv.org:1906.11516 [pdf] - 1907642
Scalability Model for the LOFAR Direction Independent Pipeline
Submitted: 2019-06-27
LOFAR is a leading aperture synthesis telescope operated in the Netherlands
with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will
produce more than 3000 14 TB data sets, mapping the entire northern sky at low
frequencies. The data produced by this survey is important for understanding
the formation and evolution of galaxies, supermassive black holes and other
astronomical phenomena. All of the LoTSS data needs to be processed by the
LOFAR Direction Independent (DI) pipeline, prefactor. Understanding the
performance of this pipeline is important when trying to optimize the
throughput for large projects, such as LoTSS and other deep surveys. Making a
model of its completion time will enable us to predict the time taken to
process large data sets, optimize our parameter choices, help schedule other
LOFAR processing services, and predict processing time for future large radio
telescopes. We tested the prefactor pipeline by scaling several parameters,
notably number of CPUs, data size and size of calibration sky model. We present
these results as a comprehensive model which will be used to predict processing
time for a wide range of processing parameters. We also discover that smaller
calibration models lead to significantly faster calibration times, while the
calibration results do not significantly degrade in quality. Finally, we
validate the model and compare predictions with production runs from the past
six months, quantifying the performance penalties incurred by processing on a
shared cluster. We conclude by noting the utility of the results and model for
the LoTSS Survey, LOFAR as a whole and for other telescopes.