The Widefield ASKAP L-band Legacy All-sky Blind surveY (WALLABY) is a next-generation survey of neutral hydrogen (HI) in the Local Universe. It uses the widefield, high-resolution capability of the Australian Square Kilometer Array Pathfinder (ASKAP), a radio interferometer consisting of 36 x 12-m dishes equipped with Phased-Array Feeds (PAFs), located in an extremely radio-quiet zone in Western Australia. WALLABY aims to survey three-quarters of the sky (-90 degr < Dec < +30 degr) to a redshift of z < 0.26, and generate spectral line image cubes at ~30 arcsec resolution and ~1.6 mJy/beam per 4 km/s channel sensitivity. ASKAP's instantaneous field of view at 1.4 GHz, delivered by the PAF's 36 beams, is about 30 sq deg. At an integrated signal-to-noise ratio of five, WALLABY is expected to detect over half a million galaxies with a mean redshift of z ~ 0.05 (~200 Mpc). The scientific goals of WALLABY include: (a) a census of gas-rich galaxies in the vicinity of the Local Group; (b) a study of the HI properties of galaxies, groups and clusters, in particular the influence of the environment on galaxy evolution; and (c) the refinement of cosmological parameters using the spatial and redshift distribution of low-bias gas-rich galaxies. For context we provide an overview of previous large-scale HI surveys. Combined with existing and new multi-wavelength sky surveys, WALLABY will enable an exciting new generation of panchromatic studies of the Local Universe. - First results from the WALLABY pilot survey are revealed, with initial data products publicly available in the CSIRO ASKAP Science Data Archive (CASDA).

We use high-resolution HI data from the WHISP survey to study the HI and angular momentum properties of a sample of 114 late-type galaxies. We explore the specific baryonic angular momentum -- baryonic mass ($j_b - M_b$) relation, and find that an unbroken power law of the form $j_b \propto M_b^{0.55 \pm 0.02}$ fits the data well, with an intrinsic scatter of $\sim 0.13 \pm 0.01$ dex. We revisit the relation between the atomic gas fraction, $f_{atm}$, and the integrated atomic stability parameter $q$ (the $f_{atm} - q$ relation), originally introduced by Obreschkow et al., and probe this parameter space by populating it with galaxies from different environments, in order to study the influence of the environment on their $j_b$, $f_{atm}$ and $q$ values. We find evidence that galaxies with close neighbours show a larger intrinsic scatter about the $f_{atm} - q$ relation compared to galaxies without close-neighbours. We also find enhanced SFR among the deviating galaxies with close neighbours. In addition, we use the bulge-to-total (B/T) ratio as a morphology proxy, and find a general trend of decreasing B/T values with increasing disc stability and HI fraction in the $f_{atm} - q$ plane, indicating a fundamental link between mass, specific angular momentum, gas fraction and morphology of galaxies.

Machine learning (automated processes that learn by example in order to classify, predict, discover or generate new data) and artificial intelligence (methods by which a computer makes decisions or discoveries that would usually require human intelligence) are now firmly established in astronomy. Every week, new applications of machine learning and artificial intelligence are added to a growing corpus of work. Random forests, support vector machines, and neural networks (artificial, deep, and convolutional) are now having a genuine impact for applications as diverse as discovering extrasolar planets, transient objects, quasars, and gravitationally-lensed systems, forecasting solar activity, and distinguishing between signals and instrumental effects in gravitational wave astronomy. This review surveys contemporary, published literature on machine learning and artificial intelligence in astronomy and astrophysics. Applications span seven main categories of activity: classification, regression, clustering, forecasting, generation, discovery, and the development of new scientific insight. These categories form the basis of a hierarchy of maturity, as the use of machine learning and artificial intelligence emerges, progresses or becomes established.

Spherical coordinate systems, which are ubiquitous in astronomy, cannot be shown without distortion on flat, two-dimensional surfaces. This poses challenges for the two complementary phases of visual exploration -- making discoveries in data by looking for relationships, patterns or anomalies -- and publication -- where the results of an exploration are made available for scientific scrutiny or communication. This is a long-standing problem, and many practical solutions have been developed. Our allskyVR approach provides a workflow for experimentation with commodity virtual reality head-mounted displays. Using the free, open source S2PLOT programming library, and the A-Frame WebVR browser-based framework, we provide a straightforward way to visualise all-sky catalogues on a user-centred, virtual celestial sphere. The allskyVR distribution contains both a quickstart option, complete with a gaze-based menu system, and a fully customisable mode for those who need more control of the immersive experience. The software is available for download from: https://github.com/cfluke/allskyVR

Visualisation of data is critical to understanding astronomical phenomena. Today, many instruments produce datasets that are too big to be downloaded to a local computer, yet many of the visualisation tools used by astronomers are deployed only on desktop computers. Cloud computing is increasingly used to provide a computation and simulation platform in astronomy, but it also offers great potential as a visualisation platform. Virtual hosted desktops, with graphics processing unit (GPU) acceleration, allow interactive, graphics-intensive desktop applications to operate co-located with astronomy datasets stored in remote data centres. By combining benchmarking and user experience testing, with a cohort of 20 astronomers, we investigate the viability of replacing physical desktop computers with virtual hosted desktops. In our work, we compare two Apple MacBook computers (one old and one new, representing hardware and opposite ends of the useful lifetime) with two virtual hosted desktops: one commercial (Amazon Web Services) and one in a private research cloud (the Australian Nectar Research Cloud). For two-dimensional image-based tasks and graphics-intensive three-dimensional operations -- typical of astronomy visualisation workflows -- we found that benchmarks do not necessarily provide the best indication of performance. When compared to typical laptop computers, virtual hosted desktops can provide a better user experience, even with lower performing graphics cards. We also found that virtual hosted desktops are equally simple to use, provide greater flexibility in choice of configuration, and may actually be a more cost-effective option for typical usage profiles.

Despite the popularity of the Graphics Processing Unit (GPU) for general purpose computing, one should not forget about the practicality of the GPU for fast scientific visualisation. As astronomers have increasing access to three dimensional (3D) data from instruments and facilities like integral field units and radio interferometers, visualisation techniques such as volume rendering offer means to quickly explore spectral cubes as a whole. As most 3D visualisation techniques have been developed in fields of research like medical imaging and fluid dynamics, many transfer functions are not optimal for astronomical data. We demonstrate how transfer functions and graphics shaders can be exploited to provide new astronomy-specific explorative colouring methods. We present 12 shaders, including four novel transfer functions specifically designed to produce intuitive and informative 3D visualisations of spectral cube data. We compare their utility to classic colour mapping. The remaining shaders highlight how common computation like filtering, smoothing and line ratio algorithms can be integrated as part of the graphics pipeline. We discuss how this can be achieved by utilising the parallelism of modern GPUs along with a shading language, letting astronomers apply these new techniques at interactive frame rates. All shaders investigated in this work are included in the open source software shwirl (Vohl, 2017).

By applying a display ecology to the {\em Deeper, Wider, Faster} proactive, simultaneous telescope observing campaign, we have shown a dramatic reduction in the time taken to inspect DECam CCD images for potential transient candidates and to produce time-critical triggers to standby telescopes. We also show how facilitating rapid corroboration of potential candidates and the exclusion of non-candidates improves the accuracy of detection; and establish that a practical and enjoyable workspace can improve the experience of an otherwise taxing task for astronomers. We provide a critical road-test of two advanced displays in a research context -- a rare opportunity to demonstrate how they can be used rather than simply discuss how they might be used to accelerate discovery.

In this data-rich era of astronomy, there is a growing reliance on automated techniques to discover new knowledge. The role of the astronomer may change from being a discoverer to being a confirmer. But what do astronomers actually look at when they distinguish between "sources" and "noise?" What are the differences between novice and expert astronomers when it comes to visual-based discovery? Can we identify elite talent or coach astronomers to maximize their potential for discovery? By looking to the field of sports performance analysis, we consider an established, domain-wide approach, where the expertise of the viewer (i.e. a member of the coaching team) plays a crucial role in identifying and determining the subtle features of gameplay that provide a winning advantage. As an initial case study, we investigate whether the SportsCode performance analysis software can be used to understand and document how an experienced HI astronomer makes discoveries in spectral data cubes. We find that the process of timeline-based coding can be applied to spectral cube data by mapping spectral channels to frames within a movie. SportsCode provides a range of easy to use methods for annotation, including feature-based codes and labels, text annotations associated with codes, and image-based drawing. The outputs, including instance movies that are uniquely associated with coded events, provide the basis for a training program or team-based analysis that could be used in unison with discipline specific analysis software. In this coordinated approach to visualization and analysis, SportsCode can act as a visual notebook, recording the insight and decisions in partnership with established analysis methods. Alternatively, in situ annotation and coding of features would be a valuable addition to existing and future visualisation and analysis packages.

Radio survey datasets comprise an increasing number of individual observations stored as sets of multidimensional data. In large survey projects, astronomers commonly face limitations regarding: 1) interactive visual analytics of sufficiently large subsets of data; 2) synchronous and asynchronous collaboration; and 3) documentation of the discovery workflow. To support collaborative data inquiry, we present encube, a large-scale comparative visual analytics framework. Encube can utilise advanced visualization environments such as the CAVE2 (a hybrid 2D and 3D virtual reality environment powered with a 100 Tflop/s GPU-based supercomputer and 84 million pixels) for collaborative analysis of large subsets of data from radio surveys. It can also run on standard desktops, providing a capable visual analytics experience across the display ecology. Encube is composed of four primary units enabling compute-intensive processing, advanced visualisation, dynamic interaction, parallel data query, along with data management. Its modularity will make it simple to incorporate astronomical analysis packages and Virtual Observatory capabilities developed within our community. We discuss how encube builds a bridge between high-end display systems (such as CAVE2) and the classical desktop, preserving all traces of the work completed on either platform -- allowing the research process to continue wherever you are.

As the quantity and resolution of spectral-cubes from optical/infrared and radio surveys increase, desktop-based visualization and analysis solutions must adapt and evolve. Novel immersive 3D environments such as the CAVE2 at Monash University can overcome personal computer's visualization limitations. CAVE2 is part advanced 2D/3D visualization space (80 stereo-capable screens providing a total of 84 million pixels) and part supercomputer ($\sim100$ TFLOPS of integrated GPU-based processing power). We present a novel visualization system enabling simultaneous 3D comparative visualization of $\sim100$ spectral-cubes. With CAVE2 augmented by our newly developed web-based controller interface, astronomers can easily organise spectral-cubes on the different display panels, apply real-time transforms to one or many spectral cubes, and request quantitative information about the displayed data. We also discuss how such a solution can help accelerate the discovery rate in varied research scenarios.

We present encube $-$ a qualitative, quantitative and comparative visualisation and analysis system, with application to high-resolution, immersive three-dimensional environments and desktop displays. encube extends previous comparative visualisation systems by considering: 1) the integration of comparative visualisation and analysis into a unified system; 2) the documentation of the discovery process; and 3) an approach that enables scientists to continue the research process once back at their desktop. Our solution enables tablets, smartphones or laptops to be used as interaction units for manipulating, organising, and querying data. We highlight the modularity of encube, allowing additional functionalities to be included as required. Additionally, our approach supports a high level of collaboration within the physical environment. We show how our implementation of encube operates in a large-scale, hybrid visualisation and supercomputing environment using the CAVE2 at Monash University, and on a local desktop, making it a versatile solution. We discuss how our approach can help accelerate the discovery rate in a variety of research scenarios.

As the volume of data grows, astronomers are increasingly faced with choices on what data to keep -- and what to throw away. Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. However, there is still a need to evaluate its potential on other type of astronomical data, such as from numerical simulations. GERLUMPH (the GPU-Enabled High Resolution cosmological MicroLensing parameter survey) represents an example of a data intensive project in theoretical astrophysics. In the next phase of processing, the ~27 terabyte GERLUMPH dataset is set to grow by a factor of 100 -- well beyond the current storage capabilities of the supercomputing facility on which it resides. In order to minimise bandwidth usage, file transfer time, and storage space, this work evaluates several data compression techniques. Specifically, we investigate off-the-shelf and custom lossless compression algorithms as well as the lossy JPEG2000 compression format. Results of lossless compression algorithms on GERLUMPH data products show small compression ratios (1.35:1 to 4.69:1 of input file size) varying with the nature of the input data. Our results suggest that JPEG2000 could be suitable for other numerical datasets stored as gridded data or volumetric data. When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. In our case study, lossy compression and a high compression ratio do not significantly compromise the intended use of the data for constraining quasar source profiles from cosmological microlensing.

Astronomical images and datasets are increasingly high-resolution and multi-dimensional. The vast majority of astronomers perform all of their visualisation and analysis tasks on low-resolution, two-dimensional desktop monitors. If there were no technological barriers to designing the ultimate stereoscopic display for astronomy, what would it look like? What capabilities would we require of our compute hardware to drive it? And are existing technologies even close to providing a true 3D experience that is compatible with the depth resolution of human stereoscopic vision? We consider the CAVE2 (an 80 Megapixel, hybrid 2D and 3D virtual reality environment directly integrated with a 100 Tflop/s GPU-powered supercomputer) and the Oculus Rift (a low- cost, head-mounted display) as examples at opposite financial ends of the immersive display spectrum.

With large-scale Integral Field Spectroscopy (IFS) surveys of thousands of galaxies currently under-way or planned, the astronomical community is in need of methods, techniques and tools that will allow the analysis of huge amounts of data. We focus on the kinematic modelling of disc galaxies and investigate the potential use of massively parallel architectures, such as the Graphics Processing Unit (GPU), as an accelerator for the computationally expensive model-fitting procedure. We review the algorithms involved in model-fitting and evaluate their suitability for GPU implementation. We employ different optimization techniques, including the Levenberg-Marquardt and Nested Sampling algorithms, but also a naive brute-force approach based on Nested Grids. We find that the GPU can accelerate the model-fitting procedure up to a factor of ~100 when compared to a single-threaded CPU, and up to a factor of ~10 when compared to a multi-threaded dual CPU configuration. Our method's accuracy, precision and robustness are assessed by successfully recovering the kinematic properties of simulated data, and also by verifying the kinematic modelling results of galaxies from the GHASP and DYNAMO surveys as found in the literature. The resulting GBKFIT code is available for download from: http://supercomputing.swin.edu.au/gbkfit.

In the upcoming synoptic all--sky survey era of astronomy, thousands of new multiply imaged quasars are expected to be discovered and monitored regularly. Light curves from the images of gravitationally lensed quasars are further affected by superimposed variability due to microlensing. In order to disentangle the microlensing from the intrinsic variability of the light curves, the time delays between the multiple images have to be accurately measured. The resulting microlensing light curves can then be analyzed to reveal information about the background source, such as the size of the quasar accretion disc. In this paper we present the most extensive and coherent collection of simulated microlensing light curves; we have generated $>2.5$ billion light curves using the GERLUMPH high resolution microlensing magnification maps. Our simulations can be used to: train algorithms to measure lensed quasar time delays, plan future monitoring campaigns, and study light curve properties throughout parameter space. Our data are openly available to the community and are complemented by online eResearch tools, located at http://gerlumph.swin.edu.au .

We derive peculiar velocities for the 6dF Galaxy Survey (6dFGS) and describe the velocity field of the nearby ($z<0.055$) southern hemisphere. The survey comprises 8885 galaxies for which we have previously reported Fundamental Plane data. We obtain peculiar velocity probability distributions for the redshift space positions of each of these galaxies using a Bayesian approach. Accounting for selection bias, we find that the logarithmic distance uncertainty is 0.11 dex, corresponding to $26\%$ in linear distance. We use adaptive kernel smoothing to map the observed 6dFGS velocity field out to $cz \sim 16,000$ \kms, and compare this to the predicted velocity fields from the PSCz Survey and the 2MASS Redshift Survey. We find a better fit to the PSCz prediction, although the reduced $\chi^2$ for the whole sample is approximately unity for both comparisons. This means that, within the observational uncertainties due to redshift independent distance errors, observed galaxy velocities and those predicted by the linear approximation from the density field agree. However, we find peculiar velocities that are systematically more positive than model predictions in the direction of the Shapley and Vela superclusters, and systematically more negative than model predictions in the direction of the Pisces-Cetus Supercluster, suggesting contributions from volumes not covered by the models.

Cosmological gravitational microlensing has been proven to be a powerful tool to constrain the structure of multiply imaged quasars, especially the accretion disc and central supermassive black-hole system. However, the derived constraints on models may be affected by large systematic errors introduced in the various stages of modelling, namely, the macromodels, the microlensing magnification maps, and the convolution with realistic disc profiles. In particular, it has been known that different macromodels of the galaxy lens that fit the observations equally well, can lead to different values of convergence, $\kappa$, and shear, $\gamma$, required to generate magnification maps. So far, 25 microlensed quasars have been studied using microlensing techniques, where each system has been modelled and analyzed individually, or in small samples. This is about to change due to the upcoming synoptic all-sky surveys, which are expected to discover thousands of quasars suitable for microlensing studies. In the present study we investigate the connection between macromodels of the galaxy lens and microlensing magnification maps throughout the parameter space in preparation for future studies of large statistical samples of systems displaying microlensing. In particular, we use 55,900 maps produced by the GERLUMPH parameter survey (available online at http://gerlumph.swin.edu.au ) and identify regions of parameter space where macromodel uncertainties ($\Delta\kappa$,$\Delta\gamma$) lead to statistically different magnification maps. Strategies for mitigating the effect of $\Delta\kappa$,$\Delta\gamma$ uncertainties are discussed in order to understand and control this potential source of systematic errors in accretion disc constraints derived from microlensing.

Clustering commodity displays into a Tiled Display Wall (TDW) provides a cost-effective way to create an extremely high resolution display, capable of approaching the image sizes now gen- erated by modern astronomical instruments. Astronomers face the challenge of inspecting single large images, many similar images simultaneously, and heterogeneous but related content. Many research institutions have constructed TDWs on the basis that they will improve the scientific outcomes of astronomical imagery. We test this concept by presenting sample images to astronomers and non- astronomers using a standard desktop display (SDD) and a TDW. These samples include standard English words, wide field galaxy surveys and nebulae mosaics from the Hubble telescope. These experiments show that TDWs provide a better environment for searching for small targets in large images than SDDs. It also shows that astronomers tend to be better at searching images for targets than non-astronomers, both groups are generally better when employing physical navigation as opposed to virtual navigation, and that the combination of two non-astronomers using a TDW rivals the experience of a single astronomer. However, there is also a large distribution in aptitude amongst the participants and the nature of the content also plays a significant role is success.

As astronomy enters the petascale data era, astronomers are faced with new challenges relating to storage, access and management of data. A shift from the traditional approach of combining data and analysis at the desktop to the use of remote services, pushing the computation to the data, is now underway. In the field of cosmological gravitational microlensing, future synoptic all--sky surveys are expected to bring the number of multiply imaged quasars from the few tens that are currently known to a few thousands. This inflow of observational data, together with computationally demanding theoretical modelling via the production of microlensing magnification maps, requires a new approach. We present our technical solutions to supporting the GPU-Enabled, High Resolution cosmological MicroLensing parameter survey (GERLUMPH). This extensive dataset for cosmological microlensing modelling comprises over 70,000 individual magnification maps and ${\sim}10^6$ related results. We describe our approaches to hosting, organizing, and serving ${\sim}$30 Terabytes of data and metadata products. We present a set of online analysis tools developed with PHP, JavaScript and WebGL to support access and analysis of GELRUMPH data in a Web browser. We discuss our use of graphics processing units (GPUs) to accelerate data production, and we release the core of the GPU-D direct inverse ray--shooting code (Thompson et al., 2010; Astrophysics Source Code Library, record ascl:1403.001) used to generate the magnification maps. All of the GERLUMPH data and tools are available online from http://gerlumph.swin.edu.au . This project made use of gSTAR, the GPU Supercomputer for Theoretical Astrophysical Research.

As synoptic all-sky surveys begin to discover new multiply lensed quasars, the flow of data will enable statistical cosmological microlensing studies of sufficient size to constrain quasar accretion disc and supermassive black hole properties. In preparation for this new era, we are undertaking the GPU-Enabled, High Resolution cosmological MicroLensing parameter survey (GERLUMPH). We present here the GERLUMPH Data Release 1, which consists of 12342 high resolution cosmological microlensing magnification maps and provides the first uniform coverage of the convergence, shear and smooth matter fraction parameter space. We use these maps to perform a comprehensive numerical investigation of the mass-sheet degeneracy, finding excellent agreement with its predictions. We study the effect of smooth matter on microlensing induced magnification fluctuations. In particular, in the minima and saddle-point regions, fluctuations are enhanced only along the critical line, while in the maxima region they are always enhanced for high smooth matter fractions (~0.9). We describe our approach to data management, including the use of an SQL database with a Web interface for data access and online analysis, obviating the need for individuals to download large volumes of data. In combination with existing observational databases and online applications, the GERLUMPH archive represents a fundamental component of a new microlensing eResearch cloud. Our maps and tools are publicly available at http://gerlumph.swin.edu.au/.

Cosmological gravitational microlensing is a useful technique for understanding the structure of the inner parts of a quasar, especially the accretion disk and the central supermassive black hole. So far, most of the cosmological microlensing studies have focused on single objects from ~90 currently known lensed quasars. However, present and planned all-sky surveys are expected to discover thousands of new lensed systems. Using a graphics processing unit (GPU) accelerated ray-shooting code, we have generated 2550 magnification maps uniformly across the convergence ({\kappa}) and shear ({\gamma}) parameter space of interest to microlensing. We examine the effect of random realizations of the microlens positions on map properties such as the magnification probability distribution (MPD). It is shown that for most of the parameter space a single map is representative of an average behaviour. All of the simulations have been carried out on the GPU-Supercomputer for Theoretical Astrophysics Research (gSTAR).

We present a high-performance, graphics processing unit (GPU)-based framework for the efficient analysis and visualization of (nearly) terabyte (TB)-sized 3-dimensional images. Using a cluster of 96 GPUs, we demonstrate for a 0.5 TB image: (1) volume rendering using an arbitrary transfer function at 7--10 frames per second; (2) computation of basic global image statistics such as the mean intensity and standard deviation in 1.7 s; (3) evaluation of the image histogram in 4 s; and (4) evaluation of the global image median intensity in just 45 s. Our measured results correspond to a raw computational throughput approaching one teravoxel per second, and are 10--100 times faster than the best possible performance with traditional single-node, multi-core CPU implementations. A scalability analysis shows the framework will scale well to images sized 1 TB and beyond. Other parallel data analysis algorithms can be added to the framework with relative ease, and accordingly, we present our framework as a possible solution to the image analysis and visualization requirements of next-generation telescopes, including the forthcoming Square Kilometre Array pathfinder radiotelescopes.

We introduce \emph{Astronomy and Computing}, a new journal for the growing population of people working in the domain where astronomy overlaps with computer science and information technology. The journal aims to provide a new communication channel within that community, which is not well served by current journals, and to help secure recognition of its true importance within modern astronomy. In this inaugural editorial, we describe the rationale for creating the journal, outline its scope and ambitions, and seek input from the community in defining in detail how the journal should work towards its high-level goals.

We present a framework to interactively volume-render three-dimensional data cubes using distributed ray-casting and volume bricking over a cluster of workstations powered by one or more graphics processing units (GPUs) and a multi-core CPU. The main design target for this framework is to provide an in-core visualization solution able to provide three-dimensional interactive views of terabyte-sized data cubes. We tested the presented framework using a computing cluster comprising 64 nodes with a total of 128 GPUs. The framework proved to be scalable to render a 204 GB data cube with an average of 30 frames per second. Our performance analyses also compare between using NVIDIA Tesla 1060 and 2050 GPU architectures and the effect of increasing the visualization output resolution on the rendering performance. Although our initial focus, and the examples presented in this work, is volume rendering of spectral data cubes from radio astronomy, we contend that our approach has applicability to other disciplines where close to real-time volume rendering of terabyte-order 3D data sets is a requirement.

Incoherent dedispersion is a computationally intensive problem that appears frequently in pulsar and transient astronomy. For current and future transient pipelines, dedispersion can dominate the total execution time, meaning its computational speed acts as a constraint on the quality and quantity of science results. It is thus critical that the algorithm be able to take advantage of trends in commodity computing hardware. With this goal in mind, we present analysis of the 'direct', 'tree' and 'sub-band' dedispersion algorithms with respect to their potential for efficient execution on modern graphics processing units (GPUs). We find all three to be excellent candidates, and proceed to describe implementations in C for CUDA using insight gained from the analysis. Using recent CPU and GPU hardware, the transition to the GPU provides a speed-up of 9x for the direct algorithm when compared to an optimised quad-core CPU code. For realistic recent survey parameters, these speeds are high enough that further optimisation is unnecessary to achieve real-time processing. Where further speed-ups are desirable, we find that the tree and sub-band algorithms are able to provide 3-7x better performance at the cost of certain smearing, memory consumption and development time trade-offs. We finish with a discussion of the implications of these results for future transient surveys. Our GPU dedispersion code is publicly available as a C library at: http://dedisp.googlecode.com/

We extend the two-dimensional Cartesian shapelet formalism to d-dimensions. Concentrating on the three-dimensional case, we derive shapelet-based equations for the mass, centroid, root-mean-square radius, and components of the quadrupole moment and moment of inertia tensors. Using cosmological N-body simulations as an application domain, we show that three-dimensional shapelets can be used to replicate the complex sub-structure of dark matter halos and demonstrate the basis of an automated classification scheme for halo shapes. We investigate the shapelet decomposition process from an algorithmic viewpoint, and consider opportunities for accelerating the computation of shapelet-based representations using graphics processing units (GPUs).

Exploration of the time-domain radio sky has huge potential for advancing our knowledge of the dynamic universe. Past surveys have discovered large numbers of pulsars, rotating radio transients and other transient radio phenomena; however, they have typically relied upon off-line processing to cope with the high data and processing rate. This paradigm rules out the possibility of obtaining high-resolution base-band dumps of significant events or of performing immediate follow-up observations, limiting analysis power to what can be gleaned from detection data alone. To overcome this limitation, real-time processing and detection of transient radio events is required. By exploiting the significant computing power of modern graphics processing units (GPUs), we are developing a transient-detection pipeline that runs in real-time on data from the Parkes radio telescope. In this paper we discuss the algorithms used in our pipeline, the details of their implementation on the GPU and the challenges posed by the presence of radio frequency interference.

Upcoming and future astronomy research facilities will systematically generate terabyte-sized data sets moving astronomy into the Petascale data era. While such facilities will provide astronomers with unprecedented levels of accuracy and coverage, the increases in dataset size and dimensionality will pose serious computational challenges for many current astronomy data analysis and visualization tools. With such data sizes, even simple data analysis tasks (e.g. calculating a histogram or computing data minimum/maximum) may not be achievable without access to a supercomputing facility. To effectively handle such dataset sizes, which exceed today's single machine memory and processing limits, we present a framework that exploits the distributed power of GPUs and many-core CPUs, with a goal of providing data analysis and visualizing tasks as a service for astronomers. By mixing shared and distributed memory architectures, our framework effectively utilizes the underlying hardware infrastructure handling both batched and real-time data analysis and visualization tasks. Offering such functionality as a service in a "software as a service" manner will reduce the total cost of ownership, provide an easy to use tool to the wider astronomical community, and enable a more optimized utilization of the underlying hardware infrastructure.

In the era of synoptic surveys, the number of known gravitationally lensed quasars is set to increase by over an order of magnitude. These new discoveries will enable a move from single-quasar studies to investigations of statistical samples, presenting new opportunities to test theoretical models for the structure of quasar accretion discs and broad emission line regions (BELRs). As one crucial step in preparing for this influx of new lensed systems, a large-scale exploration of microlensing convergence-shear parameter space is warranted, requiring the computation of O(10^5) high resolution magnification maps. Based on properties of known lensed quasars, and expectations from accretion disc/BELR modelling, we identify regions of convergence-shear parameter space, map sizes, smooth matter fractions, and pixel resolutions that should be covered. We describe how the computationally time-consuming task of producing ~290000 magnification maps with sufficient resolution (10000^2 pixel/map) to probe scales from the inner edge of the accretion disc to the BELR can be achieved in ~400 days on a 100 teraflop/s high performance computing facility, where the processing performance is achieved with graphics processing units. We illustrate a use-case for the parameter survey by investigating the effects of varying the lens macro-model on accretion disc constraints in the lensed quasar Q2237+0305. We find that although all constraints are consistent within their current error bars, models with more densely packed microlenses tend to predict shallower accretion disc radial temperature profiles. With a large parameter survey such as the one described here, such systematics on microlensing measurements could be fully explored.

In recent years, the Graphics Processing Unit (GPU) has emerged as a low-cost alternative for high performance computing, enabling impressive speed-ups for a range of scientific computing applications. Early adopters in astronomy are already benefiting in adapting their codes to take advantage of the GPU's massively parallel processing paradigm. I give an introduction to, and overview of, the use of GPUs in astronomy to date, highlighting the adoption and application trends from the first ~100 GPU-related publications in astronomy. I discuss the opportunities and challenges of utilising GPU computing clusters, such as the new Australian GPU supercomputer, gSTAR, for accelerating the rate of astronomical discovery.

High redshift sources suffer from magnification or demagnification due to weak gravitational lensing by large scale structure. One consequence of this is that the distance-redshift relation, in wide use for cosmological tests, suffers lensing-induced scatter which can be quantified by the magnification probability distribution. Predicting this distribution generally requires a method for ray-tracing through cosmological N-body simulations. However, standard methods tend to apply the multiple thin-lens approximation. In an effort to quantify the accuracy of these methods, we develop an innovative code that performs ray-tracing without the use of this approximation. The efficiency and accuracy of this computationally challenging approach can be improved by careful choices of numerical parameters; therefore, the results are analysed for the behaviour of the ray-tracing code in the vicinity of Schwarzschild and Navarro-Frenk-White lenses. Preliminary comparisons are drawn with the multiple lens-plane ray-bundle method in the context of cosmological mass distributions for a source redshift of $z_{s}=0.5$.

Astronomy is entering a new era of discovery, coincident with the establishment of new facilities for observation and simulation that will routinely generate petabytes of data. While an increasing reliance on automated data analysis is anticipated, a critical role will remain for visualization-based knowledge discovery. We have investigated scientific visualization applications in astronomy through an examination of the literature published during the last two decades. We identify the two most active fields for progress - visualization of large-N particle data and spectral data cubes - discuss open areas of research, and introduce a mapping between astronomical sources of data and data representations used in general purpose visualization tools. We discuss contributions using high performance computing architectures (e.g: distributed processing and GPUs), collaborative astronomy visualization, the use of workflow systems to store metadata about visualization parameters, and the use of advanced interaction devices. We examine a number of issues that may be limiting the spread of scientific visualization research in astronomy and identify six grand challenges for scientific visualization research in the Petascale Astronomy Era.

Flexion-based weak gravitational lensing analysis is proving to be a useful adjunct to traditional shear-based techniques. As flexion arises from gradients across an image, analytic and numerical techniques are required to investigate flexion predictions for extended image/source pairs. Using the Schwarzschild lens model, we demonstrate that the ray-bundle method for gravitational lensing can be used to accurately recover second flexion, and is consistent with recovery of zero first flexion. Using lens plane to source plane bundle propagation, we find that second flexion can be recovered with an error no worse than 1% for bundle radii smaller than {\Delta}{\theta} = 0.01 {\theta}_E and lens plane impact pararameters greater than {\theta}_E + {\Delta}{\theta}, where {\theta}_E is the angular Einstein radius. Using source plane to lens plane bundle propagation, we demonstrate the existence of a preferred flexion zone. For images at radii closer to the lens than the inner boundary of this zone, indicative of the true strong lensing regime, the flexion formalism should be used with caution (errors greater than 5% for extended image/source pairs). We also define a shear zone boundary, beyond which image shapes are essentially indistinguishable from ellipses (1% error in ellipticity). While suggestive that a traditional weak lensing analysis is satisfactory beyond this boundary, a potentially detectable non-zero flexion signal remains.

Structural parameters are normally extracted from observed galaxies by fitting analytic light profiles to the observations. Obtaining accurate fits to high-resolution images is a computationally expensive task, requiring many model evaluations and convolutions with the imaging point spread function. While these algorithms contain high degrees of parallelism, current implementations do not exploit this property. With evergrowing volumes of observational data, an inability to make use of advances in computing power can act as a constraint on scientific outcomes. This is the motivation behind our work, which aims to implement the model-fitting procedure on a graphics processing unit (GPU). We begin by analysing the algorithms involved in model evaluation with respect to their suitability for modern many-core computing architectures like GPUs, finding them to be well-placed to take advantage of the high memory bandwidth offered by this hardware. Following our analysis, we briefly describe a preliminary implementation of the model fitting procedure using freely-available GPU libraries. Early results suggest a speed-up of around 10x over a CPU implementation. We discuss the opportunities such a speed-up could provide, including the ability to use more computationally expensive but better-performing fitting routines to increase the quality and robustness of fits.

Visualisation and analysis of terabyte-scale datacubes, as will be produced with the Australian Square Kilometre Array Pathfinder (ASKAP), will pose challenges for existing astronomy software and the work practices of astronomers. Focusing on the proposed outcomes of WALLABY (Widefield ASKAP L-Band Legacy All-Sky Blind Survey), and using lessons learnt from HIPASS (HI Parkes All Sky Survey), we identify issues that astronomers will face with WALLABY data cubes. We comment on potential research directions and possible solutions to these challenges.

The Australian SKA Pathfinder (ASKAP) will be producing 2.2 terabyte HI spectral-line cubes for each 8 hours of observation by 2013. Global views of spectral data cubes are vital for the detection of instrumentation errors, the identification of data artefacts and noise characteristics, and the discovery of strange phenomena, unexpected relations, or unknown patterns. We have previously presented the first framework that can render ASKAP-sized cubes at interactive frame rates. The framework provides the user with a real-time interactive volume rendering by combining shared and distributed memory architectures, distributed CPUs and graphics processing units (GPUs), using the ray-casting algorithm. In this paper we present two main extensions of this framework which are: using a multi-panel display system to provide a high resolution rendering output, and the ability to integrate automated data analysis tools into the visualization output and to interact with its output in place.

General purpose computing on graphics processing units (GPGPU) is dramatically changing the landscape of high performance computing in astronomy. In this paper, we identify and investigate several key decision areas, with a goal of simplyfing the early adoption of GPGPU in astronomy. We consider the merits of OpenCL as an open standard in order to reduce risks associated with coding in a native, vendor-specific programming environment, and present a GPU programming philosophy based on using brute force solutions. We assert that effective use of new GPU-based supercomputing facilities will require a change in approach from astronomers. This will likely include improved programming training, an increased need for software development best-practice through the use of profiling and related optimisation tools, and a greater reliance on third-party code libraries. As with any new technology, those willing to take the risks, and make the investment of time and effort to become early adopters of GPGPU in astronomy, stand to reap great benefits.

3D visualization is an important data analysis and knowledge discovery tool, however, interactive visualization of large 3D astronomical datasets poses a challenge for many existing data visualization packages. We present a solution to interactively visualize larger-than-memory 3D astronomical data cubes by utilizing a heterogeneous cluster of CPUs and GPUs. The system partitions the data volume into smaller sub-volumes that are distributed over the rendering workstations. A GPU-based ray casting volume rendering is performed to generate images for each sub-volume, which are composited to generate the whole volume output, and returned to the user. Datasets including the HI Parkes All Sky Survey (HIPASS - 12 GB) southern sky and the Galactic All Sky Survey (GASS - 26 GB) data cubes were used to demonstrate our framework's performance. The framework can render the GASS data cube with a maximum render time < 0.3 second with 1024 x 1024 pixels output resolution using 3 rendering workstations and 8 GPUs. Our framework will scale to visualize larger datasets, even of Terabyte order, if proper hardware infrastructure is available.

Astronomy depends on ever increasing computing power. Processor clock-rates have plateaued, and increased performance is now appearing in the form of additional processor cores on a single chip. This poses significant challenges to the astronomy software community. Graphics Processing Units (GPUs), now capable of general-purpose computation, exemplify both the difficult learning-curve and the significant speedups exhibited by massively-parallel hardware architectures. We present a generalised approach to tackling this paradigm shift, based on the analysis of algorithms. We describe a small collection of foundation algorithms relevant to astronomy and explain how they may be used to ease the transition to massively-parallel computing architectures. We demonstrate the effectiveness of our approach by applying it to four well-known astronomy problems: Hogbom CLEAN, inverse ray-shooting for gravitational lensing, pulsar dedispersion and volume rendering. Algorithms with well-defined memory access patterns and high arithmetic intensity stand to receive the greatest performance boost from massively-parallel architectures, while those that involve a significant amount of decision-making may struggle to take advantage of the available processing power.

To assess how future progress in gravitational microlensing computation at high optical depth will rely on both hardware and software solutions, we compare a direct inverse ray-shooting code implemented on a graphics processing unit (GPU) with both a widely-used hierarchical tree code on a single-core CPU, and a recent implementation of a parallel tree code suitable for a CPU-based cluster supercomputer. We examine the accuracy of the tree codes through comparison with a direct code over a much wider range of parameter space than has been feasible before. We demonstrate that all three codes present comparable accuracy, and choice of approach depends on considerations relating to the scale and nature of the microlensing problem under investigation. On current hardware, there is little difference in the processing speed of the single-core CPU tree code and the GPU direct code, however the recent plateau in single-core CPU speeds means the existing tree code is no longer able to take advantage of Moore's law-like increases in processing speed. Instead, we anticipate a rapid increase in GPU capabilities in the next few years, which is advantageous to the direct code. We suggest that progress in other areas of astrophysical computation may benefit from a transition to GPUs through the use of "brute force" algorithms, rather than attempting to port the current best solution directly to a GPU language -- for certain classes of problems, the simple implementation on GPUs may already be no worse than an optimised single-core CPU version.

Astronomers have come to rely on the increasing performance of computers to reduce, analyze, simulate and visualize their data. In this environment, faster computation can mean more science outcomes or the opening up of new parameter spaces for investigation. If we are to avoid major issues when implementing codes on advanced architectures, it is important that we have a solid understanding of our algorithms. A recent addition to the high-performance computing scene that highlights this point is the graphics processing unit (GPU). The hardware originally designed for speeding-up graphics rendering in video games is now achieving speed-ups of $O(100\times)$ in general-purpose computation -- performance that cannot be ignored. We are using a generalized approach, based on the analysis of astronomy algorithms, to identify the optimal problem-types and techniques for taking advantage of both current GPU hardware and future developments in computing architectures.

Spectral datasets obtained at radio frequencies and optical/IR wavelengths are increasing in complexity as new facilities and instruments come online, resulting in an increased need to visualize and quantitatively analyze the velocity structures. As the visible structure in spectral data cubes is not purely spatial, additional insight is required to relate structures in 2D space plus line-of-sight velocity to their true three-dimensional (3D) structures. This can be achieved through the use of models that are converted to velocity-space representations. We have used the S2PLOT programming library to enable intuitive, interactive comparison between 3D models and spectral data, with potential for improved understanding of the spatial configurations. We also report on the use of 3D Cartesian shapelets to support quantitative analysis.

Traditional analysis techniques may not be sufficient for astronomers to make the best use of the data sets that current and future instruments, such as the Square Kilometre Array and its Pathfinders, will produce. By utilizing the incredible pattern-recognition ability of the human mind, scientific visualization provides an excellent opportunity for astronomers to gain valuable new insight and understanding of their data, particularly when used interactively in 3D. The goal of our work is to establish the feasibility of a real-time 3D monitoring system for data going into the Australian SKA Pathfinder archive. Based on CUDA, an increasingly popular development tool, our work utilizes the massively parallel architecture of modern graphics processing units (GPUs) to provide astronomers with an interactive 3D volume rendering for multi-spectral data sets. Unlike other approaches, we are targeting real time interactive visualization of datasets larger than GPU memory while giving special attention to data with low signal to noise ratio - two critical aspects for astronomy that are missing from most existing scientific visualization software packages. Our framework enables the astronomer to interact with the geometrical representation of the data, and to control the volume rendering process to generate a better representation of their datasets.

Gravitational lensing calculation using a direct inverse ray-shooting approach is a computationally expensive way to determine magnification maps, caustic patterns, and light-curves (e.g. as a function of source profile and size). However, as an easily parallelisable calculation, gravitational ray-shooting can be accelerated using programmable graphics processing units (GPUs). We present our implementation of inverse ray-shooting for the NVIDIA G80 generation of graphics processors using the NVIDIA Compute Unified Device Architecture (CUDA) software development kit. We also extend our code to multiple-GPU systems, including a 4-GPU NVIDIA S1070 Tesla unit. We achieve sustained processing performance of 182 Gflop/s on a single GPU, and 1.28 Tflop/s using the Tesla unit. We demonstrate that billion-lens microlensing simulations can be run on a single computer with a Tesla unit in timescales of order a day without the use of a hierarchical tree code.

Flexion is a non-linear gravitational lensing effect that arises from gradients in the convergence and shear across an image. We derive a formalism that describes non-linear gravitational lensing by a circularly symmetric lens in the thin-lens approximation. This provides us with relatively simple expressions for first- and second-flexion in terms of only the surface density and projected mass distribution of the lens. We give details of exact lens models, in particular providing flexion calculations for a Sersic-law profile, which has become increasingly popular over recent years. We further provide a single resource for the analytic forms of convergence, shear, first- and second-flexion for the following mass distributions: a point mass, singular isothermal sphere (SIS); Navarro-Frenk-White (NFW) profile; Sersic-law profile. We quantitatively compare these mass distributions and show that the convergence and first-flexion are better indicators of the Sersic shape parameter, while for the concentration of NFW profiles the shear and second-flexion terms are preferred.

We demonstrate how interactive, three-dimensional (3-d) scientific visualizations can be efficiently interchanged between a variety of mediums. Through the use of an appropriate interchange format, and a unified interaction interface, we minimize the effort to produce visualizations appropriate for undertaking knowledge discovery at the astronomer's desktop, as part of conference presentations, in digital publications or as Web content. We use examples from cosmological visualization to address some of the issues of interchange, and to describe our approach to adapting S2PLOT desktop visualizations to the Web. Supporting demonstrations are available at http://astronomy.swin.edu.au/s2plot/interchange/

Most research data collections created or used by astronomers are intrinsically multi-dimensional. In contrast, all visual representations of data presented within research papers are exclusively 2-dimensional. We present a resolution of this dichotomy that uses a novel technique for embedding 3-dimensional (3-d) visualisations of astronomy data sets in electronic-format research papers. Our technique uses the latest Adobe Portable Document Format extensions together with a new version of the S2PLOT programming library. The 3-d models can be easily rotated and explored by the reader and, in some cases, modified. We demonstrate example applications of this technique including: 3-d figures exhibiting subtle structure in redshift catalogues, colour-magnitude diagrams and halo merger trees; 3-d isosurface and volume renderings of cosmological simulations; and 3-d models of instructional diagrams and instrument designs.

We present a new, three-dimensional (3D) plotting library with advanced features, and support for standard and enhanced display devices. The library - S2PLOT - is written in C and can be used by C, C++ and FORTRAN programs on GNU/Linux and Apple/OSX systems. S2PLOT draws objects in a 3D (x,y,z) Cartesian space and the user interactively controls how this space is rendered at run time. With a PGPLOT inspired interface, S2PLOT provides astronomers with elegant techniques for displaying and exploring 3D data sets directly from their program code, and the potential to use stereoscopic and dome display devices. The S2PLOT architecture supports dynamic geometry and can be used to plot time-evolving data sets, such as might be produced by simulation codes. In this paper, we introduce S2PLOT to the astronomical community, describe its potential applications, and present some example uses of the library.

Despite the large budgets spent annually on astronomical research equipment such as telescopes, instruments and supercomputers, the general trend is to analyse and view the resulting datasets using small, two-dimensional displays. We report here on alternative advanced image displays, with an emphasis on displays that we have constructed, including stereoscopic projection, multiple projector tiled displays and a digital dome. These displays can provide astronomers with new ways of exploring the terabyte and petabyte datasets that are now regularly being produced from all-sky surveys, high-resolution computer simulations, and Virtual Observatory projects. We also present a summary of the Advanced Image Displays for Astronomy (AIDA) survey which we conducted from March-May 2005, in order to raise some issues pertitent to the current and future level of use of advanced image displays.

We use published mid-IR and V-band flux ratios for images A and B of Q2237+0305 to demonstrate that the size of the mid-IR emission region has a scale comparable to or larger than the microlens Einstein Radius (ER, ~10^17 cm for solar mass stars). Q2237+0305 has been monitored extensively in the R and V-bands for ~15 years. The variability record shows significant microlensing variability of the optical emission region, and has been used by several studies to demonstrate that the optical emission region is much smaller than the ER for solar-mass objects. For the majority of the monitoring history, the optical flux ratios have differed significantly from those predicted by macro-models. In contrast, recent observations in mid-IR show flux ratios similar to those measured in the radio, and to predictions of some lens models, implying that the mid-IR flux is emitted from a region that is at least 2 orders of magnitude larger than the optical emission region. We have calculated the likeli-hood of the observed mid-IR flux ratio as a function of mid-IR source size given the observed V-band flux ratio. The expected flux ratio for a source having dimensions of ~1 ER is a sensitive function of the macro model adopted. However we find that the probability of source size given the observed flux ratios is primarily sensitive to the ratio of the macro-model magnifications. The majority of published macro models for Q2237+0305 yield a flux ratio for images B and A of 0.8 - 1.1. By combining probabilities from the ratios A/B and C/D we infer that the diameter of a circular IR emission region is >1ER with >95% confidence. For microlensing by low-mass stars, this source size limit rules out non-thermal processes such as synchrotron as mechanisms for mid-IR emission.

We present the results of weak gravitational lensing statistics in four different cosmological $N$-body simulations. The data has been generated using an algorithm for the three-dimensional shear, which makes use of a variable softening facility for the $N$-body particle masses, and enables a physical interpretation for the large-scale structure to be made. Working in three-dimensions also allows the correct use of the appropriate angular diameter distances. Our results are presented on the basis of the filled beam approximation in view of the variable particle softening scheme in our algorithm. The importance of the smoothness of matter in the universe for the weak lensing results is discussed in some detail. The low density cosmology with a cosmological constant appears to give the broadest distributions for all the statistics computed for sources at high redshifts. In particular, the range in magnification values for this cosmology has implications for the determination of the cosmological parameters from high-redshift Type Ia Supernov\ae. The possibility of determining the density parameter from the non-Gaussianity in the probability distribution for the convergence is discussed.

We present here an alternative method for calculating magnifications in gravitational lensing calculations -- the Ray Bundle method. We provide a detailed comparison between the distribution of magnifications obtained compared with analytic results and conventional ray-shooting methods. The Ray Bundle method provides high accuracy in the weak lensing limit, and is computationally much faster than (non-hierarchical) ray shooting methods to a comparable accuracy. The Ray Bundle method is a powerful and efficient technique with which to study gravitational lensing within realistic cosmological models, particularly in the weak lensing limit.