Normalized to: Lezzi, D.
[1]
oai:arXiv.org:1704.05064 [pdf] - 1564182
Calibration of LOFAR data on the cloud
Submitted: 2017-04-17
New scientific instruments are starting to generate an unprecedented amount
of data. LOFAR, one of the Square Kilometre Array pathfinders, is already
producing data on a petabyte scale. The calibration of these data presents a
huge challenge for final users: a) extensive storage and computing resources
are required; b) the installation and maintenance of the processing software is
not trivial; and c) the requirements of (experimental) calibration pipelines
are quickly evolving. After encountering some limitations in classical
infrastructures, we investigated the viability of cloud infrastructures as a
solution. We found that the installation and operation of LOFAR data
calibration pipelines is not only possible, but can also be efficient in cloud
infrastructures. The main advantages were: (1) ease of software installation
and maintenance, and the availability of standard APIs and tools, widely used
in the industry; this reduces the requirement for significant manual
intervention, which can have a highly negative impact; (2) flexibility to adapt
the infrastructure to the needs of the problem, especially as those demands
change over time; (3) on-demand consumption of (shared) resources. We found no
significant impediments associated with the speed of data transfer, the use of
external block storage, or the memory available. However, the availability of
scratch storage areas of an appropriate size is critical. Finally, we
considered the cost-effectiveness of a commercial cloud like Amazon Web
Services. While it is more expensive than the operation of a large,
fully-utilised cluster completely dedicated to LOFAR data reduction, its costs
are competitive if the number of datasets to be analysed is not high, or if the
costs of maintaining the dedicated system become high. Coupled with the
advantages discussed above, this suggests that a cloud infrastructure may be
favourable for many users.