Normalized to: Doyle, P.
[1]
oai:arXiv.org:1502.02821 [pdf] - 933123
Building a scalable global data processing pipeline for large
astronomical photometric datasets
Submitted: 2015-02-10
Astronomical photometry is the science of measuring the flux of a celestial
object. Since its introduction, the CCD has been the principle method of
measuring flux to calculate the apparent magnitude of an object. Each CCD image
taken must go through a process of cleaning and calibration prior to its use.
As the number of research telescopes increases the overall computing resources
required for image processing also increases. Existing processing techniques
are primarily sequential in nature, requiring increasingly powerful servers,
faster disks and faster networks to process data. Existing High Performance
Computing solutions involving high capacity data centres are complex in design
and expensive to maintain, while providing resources primarily to high profile
science projects. This research describes three distributed pipeline
architectures, a virtualised cloud based IRAF, the Astronomical Compute Node
(ACN), a private cloud based pipeline, and NIMBUS, a globally distributed
system. The ACN pipeline processed data at a rate of 4 Terabytes per day
demonstrating data compression and upload to a central cloud storage service at
a rate faster than data generation. The primary contribution of this research
is NIMBUS, which is rapidly scalable, resilient to failure and capable of
processing CCD image data at a rate of hundreds of Terabytes per day. This
pipeline is implemented using a decentralised web queue to control the
compression of data, uploading of data to distributed web servers, and creating
web messages to identify the location of the data. Using distributed web queue
messages, images are downloaded by computing resources distributed around the
globe. Rigorous experimental evidence is presented verifying the horizontal
scalability of the system which has demonstrated a processing rate of 192
Terabytes per day with clear indications that higher processing rates are
possible.
[2]
oai:arXiv.org:astro-ph/0009202 [pdf] - 38044
ASCA and other contemporaneous observations of the blazar B2 1308+326
Watson, D.;
Smith, N.;
Hanlon, L.;
McBreen, B.;
Quilligan, F.;
Tashiro, M.;
Metcalfe, L.;
Doyle, P.;
Teraesranta, H.;
Carraminana, A.;
Guichard, J.
Submitted: 2000-09-13
The high redshift (z=0.997) blazar B2 1308+326 was observed contemporaneously
at x-ray, optical and radio wavelengths in June 1996. The x-ray observations
were performed with ASCA. The ASCA results were found to be consistent with
reanalysed data from two earlier ROSAT observations. The combined ASCA and
ROSAT data reveal an x-ray spectrum that is best fit by a broken power law with
absorber model. The break in the x-ray spectrum is interpreted, to be the
emerging importance of inverse Compton (IC) emission which dominates the ASCA
spectrum. The faint optical state reported for these observations
(m_V=18.3+/-0.25) is incompatible with the high synchrotron flux previously
detected by ROSAT. The IC emission detected by both ROSAT and ASCA was not
significantly affected by the large change in the synchrotron component. MgII
emission was detected with an equivalent width (EW) significantly different
from previously reported values. Absorption at a level of in excess of the
Galactic value was detected, indicating the possible presence of a foreground
absorber. A gravitational microlensing scenario cannot therefore be ruled out
for this blazar. B2 1308+326 could be a typical radio-selected BL Lac in terms
of peak synchrotron frequency and optical and radio variability but its high
bolometric luminosity, variable line emission and high Doppler boost factor
make it appear more like a quasar than a BL Lac. It is suggested that B2
1308+326 be considered as the prototype of this class of composite source.