We present an extension to the model-free anomaly detection algorithm, Isolation Forest. This extension, named Extended Isolation Forest (EIF), resolves issues with assignment of anomaly score to given data points. We motivate the problem using heat maps for anomaly scores. These maps suffer from artifacts generated by the criteria for branching operation of the binary tree. We explain this problem in detail and demonstrate the mechanism by which it occurs visually. We then propose two different approaches for improving the situation. First we propose transforming the data randomly before creation of each tree, which results in averaging out the bias. Second, which is the preferred way, is to allow the slicing of the data to use hyperplanes with random slopes. This approach results in remedying the artifact seen in the anomaly score heat maps. We show that the robustness of the algorithm is much improved using this method by looking at the variance of scores of data points distributed along constant level sets. We report AUROC and AUPRC for our synthetic datasets, along with real-world benchmark datasets. We find no appreciable difference in the rate of convergence nor in computation time between the standard Isolation Forest and EIF.

We perform a comparison of different approaches to star-galaxy classification using the broad-band photometric data from Year 1 of the Dark Energy Survey. This is done by performing a wide range of tests with and without external `truth' information, which can be ported to other similar datasets. We make a broad evaluation of the performance of the classifiers in two science cases with DES data that are most affected by this systematic effect: large-scale structure and Milky Way studies. In general, even though the default morphological classifiers used for DES Y1 cosmology studies are sufficient to maintain a low level of systematic contamination from stellar mis-classification, contamination can be reduced to the O(1%) level by using multi-epoch and infrared information from external datasets. For Milky Way studies the stellar sample can be augmented by ~20% for a given flux limit. Reference catalogs used in this work will be made available upon publication.

The ever-growing datasets in observational astronomy have challenged scientists in many aspects, including an efficient and interactive data exploration and visualization. Many tools have been developed to confront this challenge. However, they usually focus on displaying the actual images or focus on visualizing patterns within catalogs in a predefined way. In this paper we introduce Vizic, a Python visualization library that builds the connection between images and catalogs through an interactive map of the sky region. Vizic visualizes catalog data over a custom background canvas using the shape, size and orientation of each object in the catalog. The displayed objects in the map are highly interactive and customizable comparing to those in the images. These objects can be filtered by or colored by their properties, such as redshift and magnitude. They also can be sub-selected using a lasso-like tool for further analysis using standard Python functions from inside a Jupyter notebook. Furthermore, Vizic allows custom overlays to be appended dynamically on top of the sky map. We have initially implemented several overlays, namely, Voronoi, Delaunay, Minimum Spanning Tree and HEALPix grid layers, which are helpful for visualizing large-scale structure. All these overlays can be generated, added or removed interactively with one line of code. The catalog data is stored in a non-relational database, and the interfaces were developed in JavaScript and Python to work within Jupyter Notebook, which allows to create custom widgets, user generated scripts to analyze and plot the data selected/displayed in the interactive map. This unique design makes Vizic a very powerful and flexible interactive analysis tool. Vizic can be adopted in variety of exercises, for example, data inspection, clustering analysis, galaxy alignment studies, outlier identification or simply large-scale visualizations.

Most existing star-galaxy classifiers use the reduced summary information from catalogs, requiring careful feature extraction and selection. The latest advances in machine learning that use deep convolutional neural networks allow a machine to automatically learn the features directly from data, minimizing the need for input from human experts. We present a star-galaxy classification framework that uses deep convolutional neural networks (ConvNets) directly on the reduced, calibrated pixel values. Using data from the Sloan Digital Sky Survey (SDSS) and the Canada-France-Hawaii Telescope Lensing Survey (CFHTLenS), we demonstrate that ConvNets are able to produce accurate and well-calibrated probabilistic classifications that are competitive with conventional machine learning techniques. Future advances in deep learning may bring more success with current and forthcoming photometric surveys, such as the Dark Energy Survey (DES) and the Large Synoptic Survey Telescope (LSST), because deep neural networks require very little, manual feature engineering.

This overview article describes the legacy prospect and discovery potential of the Dark Energy Survey (DES) beyond cosmological studies, illustrating it with examples from the DES early data. DES is using a wide-field camera (DECam) on the 4m Blanco Telescope in Chile to image 5000 sq deg of the sky in five filters (grizY). By its completion the survey is expected to have generated a catalogue of 300 million galaxies with photometric redshifts and 100 million stars. In addition, a time-domain survey search over 27 sq deg is expected to yield a sample of thousands of Type Ia supernovae and other transients. The main goals of DES are to characterise dark energy and dark matter, and to test alternative models of gravity; these goals will be pursued by studying large scale structure, cluster counts, weak gravitational lensing and Type Ia supernovae. However, DES also provides a rich data set which allows us to study many other aspects of astrophysics. In this paper we focus on additional science with DES, emphasizing areas where the survey makes a difference with respect to other current surveys. The paper illustrates, using early data (from `Science Verification', and from the first, second and third seasons of observations), what DES can tell us about the solar system, the Milky Way, galaxy evolution, quasars, and other topics. In addition, we show that if the cosmological model is assumed to be Lambda+ Cold Dark Matter (LCDM) then important astrophysics can be deduced from the primary DES probes. Highlights from DES early data include the discovery of 34 Trans Neptunian Objects, 17 dwarf satellites of the Milky Way, one published z > 6 quasar (and more confirmed) and two published superluminous supernovae (and more confirmed).

Photometric surveys produce large-area maps of the galaxy distribution, but with less accurate redshift information than is obtained from spectroscopic methods. Modern photometric redshift (photo-z) algorithms use galaxy magnitudes, or colors, that are obtained through multi-band imaging to produce a probability density function (PDF) for each galaxy in the map. We used simulated data to study the effect of using different photo-z estimators to assign galaxies to redshift bins in order to compare their effects on angular clustering and galaxy bias measurements. We found that if we use the entire PDF, rather than a single-point (mean or mode) estimate, the deviations are less biased, especially when using narrow redshift bins. When the redshift bin widths are $\Delta z=0.1$, the use of the entire PDF reduces the typical measurement bias from 5%, when using single point estimates, to 3%.

We extend a machine learning (ML) framework presented previously to model galaxy formation and evolution in a hierarchical universe using N-body + hydrodynamical simulations. In this work, we show that ML is a promising technique to study galaxy formation in the backdrop of a hydrodynamical simulation. We use the Illustris Simulation to train and test various sophisticated machine learning algorithms. By using only essential dark matter halo physical properties and no merger history, our model predicts the gas mass, stellar mass, black hole mass, star formation rate, $g-r$ color, and stellar metallicity fairly robustly. Our results provide a unique and powerful phenomenological framework to explore the galaxy-halo connection that is built upon a solid hydrodynamical simulation. The promising reproduction of the listed galaxy properties demonstrably place ML as a promising and a significantly more computationally efficient tool to study small-scale structure formation. We find that ML mimics a full-blown hydrodynamical simulation surprisingly well in a computation time of mere minutes. The population of galaxies simulated by ML, while not numerically identical to Illustris, is statistically and physically robust and follows the same fundamental observational constraints. Machine learning offers an intriguing and promising technique to create quick mock galaxy catalogs in the future.

We study the clustering of galaxies detected at $i<22.5$ in the Science Verification observations of the Dark Energy Survey (DES). Two-point correlation functions are measured using $2.3\times 10^6$ galaxies over a contiguous 116 deg$^2$ region in five bins of photometric redshift width $\Delta z = 0.2$ in the range $0.2 < z < 1.2.$ The impact of photometric redshift errors are assessed by comparing results using a template-based photo-$z$ algorithm (BPZ) to a machine-learning algorithm (TPZ). A companion paper (Leistedt et al 2015) presents maps of several observational variables (e.g. seeing, sky brightness) which could modulate the galaxy density. Here we characterize and mitigate systematic errors on the measured clustering which arise from these observational variables, in addition to others such as Galactic dust and stellar contamination. After correcting for systematic effects we measure galaxy bias over a broad range of linear scales relative to mass clustering predicted from the Planck $\Lambda$CDM model, finding agreement with CFHTLS measurements with $\chi^2$ of 4.0 (8.7) with 5 degrees of freedom for the TPZ (BPZ) redshifts. We test a "linear bias" model, in which the galaxy clustering is a fixed multiple of the predicted non-linear dark-matter clustering. The precision of the data allow us to determine that the linear bias model describes the observed galaxy clustering to $2.5\%$ accuracy down to scales at least $4$ to $10$ times smaller than those on which linear theory is expected to be sufficient.

We report the observation and confirmation of the first group- and cluster-scale strong gravitational lensing systems found in Dark Energy Survey (DES) data. Through visual inspection of data from the Science Verification (SV) season, we identified 53 candidate systems. We then obtained spectroscopic follow-up of 21 candidates using the Gemini Multi-Object Spectrograph (GMOS) at the Gemini South telescope and the Inamori-Magellan Areal Camera and Spectrograph (IMACS) at the Magellan/Baade telescope. With this follow-up, we confirmed six candidates as gravitational lenses: Three of the systems are newly discovered, and the remaining three were previously known. Of the 21 observed candidates, the remaining 15 were either not detected in spectroscopic observations, were observed and did not exhibit continuum emission (or spectral features), or were ruled out as lensing systems. The confirmed sample consists of one group-scale and five galaxy cluster-scale lenses. The lensed sources range in redshift z ~ 0.80-3.2, and in i-band surface brightness i_{SB} ~ 23-25 mag/sq.-arcsec. (2" aperture). For each of the six systems, we estimate the Einstein radius and the enclosed mass, which have ranges ~ 5.0 - 8.6" and ~ 7.5 x 10^{12} - 6.4 x 10^{13} solar masses, respectively.

The Third Reference Catalogue of Bright Galaxies (RC3) is a reasonably complete listing of 23,011 nearby, large, bright galaxies. By using the final imaging data release from the Sloan Digital Sky Survey, we generate scientifically-calibrated FITS mosaics by using the montage program for all SDSS imaging bands for all RC3 galaxies that lie within the survey footprint. We further combine the SDSS g, r, and i band FITS mosaics for these galaxies to create color-composite images by using the STIFF program. We generalized this software framework to make FITS mosaics and color-composite images for an arbitrary catalog and imaging data set. Due to positional inaccuracies inherent in the RC3 catalog, we employ a recursive algorithm in our mosaicking pipeline that first determines the correct location for each galaxy, and subsequently applies the mosaicking procedure. As an additional test of this new software pipeline and to obtain mosaic images of a larger sample of RC3 galaxies, we also applied this pipeline to photographic data taken by the Second Palomar Observatory Sky Survey with $B_J$, $R_F$, and $I_N$ plates. We publicly release all generated data, accessible via a web search form, and the software pipeline to enable others to make galaxy mosaics by using other catalogs or surveys.

We present a new exploratory framework to model galaxy formation and evolution in a hierarchical universe by using machine learning (ML). Our motivations are two-fold: (1) presenting a new, promising technique to study galaxy formation, and (2) quantitatively analyzing the extent of the influence of dark matter halo properties on galaxies in the backdrop of semi-analytical models (SAMs). We use the influential Millennium Simulation and the corresponding Munich SAM to train and test various sophisticated machine learning algorithms (k-Nearest Neighbors, decision trees, random forests and extremely randomized trees). By using only essential dark matter halo physical properties for haloes of $M>10^{12} M_{\odot}$ and a partial merger tree, our model predicts the hot gas mass, cold gas mass, bulge mass, total stellar mass, black hole mass and cooling radius at z = 0 for each central galaxy in a dark matter halo for the Millennium run. Our results provide a unique and powerful phenomenological framework to explore the galaxy-halo connection that is built upon SAMs and demonstrably place ML as a promising and a computationally efficient tool to study small-scale structure formation.

There exist a variety of star-galaxy classification techniques, each with their own strengths and weaknesses. In this paper, we present a novel meta-classification framework that combines and fully exploits different techniques to produce a more robust star-galaxy classification. To demonstrate this hybrid, ensemble approach, we combine a purely morphological classifier, a supervised machine learning method based on random forest, an unsupervised machine learning method based on self-organizing maps, and a hierarchical Bayesian template fitting method. Using data from the CFHTLenS survey, we consider different scenarios: when a high-quality training set is available with spectroscopic labels from DEEP2, SDSS, VIPERS, and VVDS, and when the demographics of sources in a low-quality training set do not match the demographics of objects in the test data set. We demonstrate that our Bayesian combination technique improves the overall performance over any individual classification method in these scenarios. Thus, strategies that combine the predictions of different classifiers may prove to be optimal in currently ongoing and forthcoming photometric surveys, such as the Dark Energy Survey and the Large Synoptic Survey Telescope.

We present results from a study of the photometric redshift performance of the Dark Energy Survey (DES), using the early data from a Science Verification (SV) period of observations in late 2012 and early 2013 that provided science-quality images for almost 200 sq.~deg.~at the nominal depth of the survey. We assess the photometric redshift performance using about 15000 galaxies with spectroscopic redshifts available from other surveys. These galaxies are used, in different configurations, as a calibration sample, and photo-$z$'s are obtained and studied using most of the existing photo-$z$ codes. A weighting method in a multi-dimensional color-magnitude space is applied to the spectroscopic sample in order to evaluate the photo-$z$ performance with sets that mimic the full DES photometric sample, which is on average significantly deeper than the calibration sample due to the limited depth of spectroscopic surveys. Empirical photo-$z$ methods using, for instance, Artificial Neural Networks or Random Forests, yield the best performance in the tests, achieving core photo-$z$ resolutions $\sigma_{68} \sim 0.08$. Moreover, the results from most of the codes, including template fitting methods, comfortably meet the DES requirements on photo-$z$ performance, therefore, providing an excellent precedent for future DES data sets.

We analyze the clustering of photometrically selected galaxy pairs by using the halo-occupation distribution (HOD) model. We measure the angular two-point auto-correlation function, $\omega(\theta)$, for galaxies and galaxy pairs in three volume-limited samples and develop an HOD to model their clustering. Our results are successfully fit by these HOD models, and we see the separation of "1-halo" and "2-halo" clustering terms for both single galaxies and galaxy pairs. Our clustering measurements and HOD model fits for the single galaxy samples are consistent with previous results. We find that the galaxy pairs generally have larger clustering amplitudes than single galaxies, and the quantities computed during the HOD fitting, e.g., effective halo mass, $M_{eff}$, and linear bias, $b_{g}$, are also larger for galaxy pairs. We find that the central fractions for galaxy pairs are significantly higher than single galaxies, which confirms that galaxy pairs are formed at the center of more massive dark matter haloes. We also model the clustering dependence of the galaxy pair correlation function on redshift, galaxy type, and luminosity. We find early-early pairs (bright galaxy pairs) cluster more strongly than late-late pairs (dim galaxy pairs), and that the clustering does not depend on the luminosity contrast between the two galaxies in the compact group.

The estimation and utilization of photometric redshift probability density functions (photo-$z$ PDFs) has become increasingly important over the last few years and currently there exist a wide variety of algorithms to compute photo-$z$'s, each with their own strengths and weaknesses. In this paper, we present a novel and efficient Bayesian framework that combines the results from different photo-$z$ techniques into a more powerful and robust estimate by maximizing the information from the photometric data. To demonstrate this we use a supervised machine learning technique based on random forest, an unsupervised method based on self-organizing maps, and a standard template fitting method but can be easily extend to other existing techniques. We use data from the DEEP2 and the SDSS surveys to explore different methods for combining the predictions from these techniques. By using different performance metrics, we demonstrate that we can improve the accuracy of our final photo-$z$ estimate over the best input technique, that the fraction of outliers is reduced, and that the identification of outliers is significantly improved when we apply a Na\"{\i}ve Bayes Classifier to this combined information. Our more robust and accurate photo-$z$ PDFs will allow even more precise cosmological constraints to be made by using current and future photometric surveys. These improvements are crucial as we move to analyze photometric data that push to or even past the limits of the available training data, which will be the case with the Large Synoptic Survey Telescope.

One of the consequences of entering the era of precision cosmology is the widespread adoption of photometric redshift probability density functions (PDFs). Both current and future photometric surveys are expected to obtain images of billions of distinct galaxies. As a result, storing and analyzing all of these PDFs will be non-trivial and even more severe if a survey plans to compute and store multiple different PDFs. In this paper we propose the use of a sparse basis representation to fully represent individual photo-$z$ PDFs. By using an Orthogonal Matching Pursuit algorithm and a combination of Gaussian and Voigt basis functions, we demonstrate how our approach is superior to a multi-Gaussian fitting, as we require approximately half of the parameters for the same fitting accuracy with the additional advantage that an entire PDF can be stored by using a 4-byte integer per basis function, and we can achieve better accuracy by increasing the number of bases. By using data from the CFHTLenS, we demonstrate that only ten to twenty points per galaxy are sufficient to reconstruct both the individual PDFs and the ensemble redshift distribution, $N(z)$, to an accuracy of 99.9% when compared to the one built using the original PDFs computed with a resolution of $\delta z = 0.01$, reducing the required storage of two hundred original values by a factor of ten to twenty. Finally, we demonstrate how this basis representation can be directly extended to a cosmological analysis, thereby increasing computational performance without losing resolution nor accuracy.

In this paper we explore the applicability of the unsupervised machine learning technique of Self Organizing Maps (SOM) to estimate galaxy photometric redshift probability density functions (PDFs). This technique takes a spectroscopic training set, and maps the photometric attributes, but not the redshifts, to a two dimensional surface by using a process of competitive learning where neurons compete to more closely resemble the training data multidimensional space. The key feature of a SOM is that it retains the topology of the input set, revealing correlations between the attributes that are not easily identified. We test three different 2D topological mapping: rectangular, hexagonal, and spherical, by using data from the DEEP2 survey. We also explore different implementations and boundary conditions on the map and also introduce the idea of a random atlas where a large number of different maps are created and their individual predictions are aggregated to produce a more robust photometric redshift PDF. We also introduced a new metric, the $I$-score, which efficiently incorporates different metrics, making it easier to compare different results (from different parameters or different photometric redshift codes). We find that by using a spherical topology mapping we obtain a better representation of the underlying multidimensional topology, which provides more accurate results that are comparable to other, state-of-the-art machine learning algorithms. Our results illustrate that unsupervised approaches have great potential for many astronomical problems, and in particular for the computation of photometric redshifts.

Large sets of objects with spectroscopic redshift measurements will be needed for imaging dark energy experiments to achieve their full potential, serving two goals:_training_, i.e., the use of objects with known redshift to develop and optimize photometric redshift algorithms; and_calibration_, i.e., the characterization of moments of redshift (or photo-z error) distributions. Better training makes cosmological constraints from a given experiment stronger, while highly-accurate calibration is needed for photo-z systematics not to dominate errors. In this white paper, we investigate the required scope of spectroscopic datasets which can serve both these purposes for ongoing and next-generation dark energy experiments, as well as the time required to obtain such data with instruments available in the next decade. Large time allocations on kilo-object spectrographs will be necessary, ideally augmented by infrared spectroscopy from space. Alternatively, precision calibrations could be obtained by measuring cross-correlation statistics using samples of bright objects from a large baryon acoustic oscillation experiment such as DESI. We also summarize the additional work on photometric redshift methods needed to prepare for ongoing and future dark energy experiments.

We present the results from a time domain study of absorption lines detected in quasar spectra with repeat observations from the Sloan Digital Sky Survey Data Release 7 (SDSS DR7). Beginning with over 4500 unique time separation baselines of various absorption line species identified in the SDSS DR7 quasar spectra, we create a catalogue of 2522 quasar absorption line systems with two to eight repeat observations, representing the largest collection of unbiased and homogeneous multi-epoch absorption systems ever published. To investigate these systems for time variability of narrow absorption lines, we refine this sample based on the reliability of the system detection, the proximity of pixels with bright sky contamination to individual absorption lines, and the quality of the continuum fit. Variability measurements of this sub-sample based on the absorption line equivalent widths yield a total of 33 systems with indications of significantly variable absorption strengths on time-scales ranging from one day to several years in the rest frame of the absorption system. Of these, at least 10 are from a class known as intervening absorption systems caused by foreground galaxies along the line of sight to the background quasar. This is the first evidence of possible absorption line variability detected in intervening systems, and their short time-scale variations suggest that small-scale structures (~10-100 au) are likely to exist in their host foreground galaxies.

Strong lensing gravitational time delays are a powerful and cost effective probe of dark energy. Recent studies have shown that a single lens can provide a distance measurement with 6-7 % accuracy (including random and systematic uncertainties), provided sufficient data are available to determine the time delay and reconstruct the gravitational potential of the deflector. Gravitational-time delays are a low redshift (z~0-2) probe and thus allow one to break degeneracies in the interpretation of data from higher-redshift probes like the cosmic microwave background in terms of the dark energy equation of state. Current studies are limited by the size of the sample of known lensed quasars, but this situation is about to change. Even in this decade, wide field imaging surveys are likely to discover thousands of lensed quasars, enabling the targeted study of ~100 of these systems and resulting in substantial gains in the dark energy figure of merit. In the next decade, a further order of magnitude improvement will be possible with the 10000 systems expected to be detected and measured with LSST and Euclid. To fully exploit these gains, we identify three priorities. First, support for the development of software required for the analysis of the data. Second, in this decade, small robotic telescopes (1-4m in diameter) dedicated to monitoring of lensed quasars will transform the field by delivering accurate time delays for ~100 systems. Third, in the 2020's, LSST will deliver 1000's of time delays; the bottleneck will instead be the aquisition and analysis of high resolution imaging follow-up. Thus, the top priority for the next decade is to support fast high resolution imaging capabilities, such as those enabled by the James Webb Space Telescope and next generation adaptive optics systems on large ground based telescopes.

With the growth of large photometric surveys, accurately estimating photometric redshifts, preferably as a probability density function (PDF), and fully understanding the implicit systematic uncertainties in this process has become increasingly important. In this paper, we present a new, publicly available, parallel, machine learning algorithm that generates photometric redshift PDFs by using prediction trees and random forest techniques, which we have named TPZ. This new algorithm incorporates measurement errors into the calculation while also dealing efficiently with missing values in the data. In addition, our implementation of this algorithm provides supplementary information regarding the data being analyzed, including unbiased estimates of the accuracy of the technique without resorting to a validation data set, identification of poor photometric redshift areas within the parameter space occupied by the spectroscopic training data, a quantification of the relative importance of the variables used to construct the PDF, and a robust identification of outliers. This extra information can be used to optimally target new spectroscopic observations and to improve the overall efficacy of the redshift estimation. We have tested TPZ on galaxy samples drawn from the SDSS main galaxy sample and from the DEEP2 survey, obtaining excellent results in each case. We also have tested our implementation by participating in the PHAT1 project, which is a blind photometric redshift contest, finding that TPZ performs comparable to if not better than other empirical photometric redshift algorithms. Finally, we discuss the various parameters that control the operation of TPZ, the specific limitations of this approach and an application of photometric redshift PDFs.

We present the galaxy two-point angular correlation function for galaxies selected from the seventh data release of the Sloan Digital Sky Survey. The galaxy sample was selected with $r$-band apparent magnitudes between 17 and 21; and we measure the correlation function for the full sample as well as for the four magnitude ranges: 17-18, 18-19, 19-20, and 20-21. We update the flag criteria to select a clean galaxy catalog and detail specific tests that we perform to characterize systematic effects, including the effects of seeing, Galactic extinction, and the overall survey uniformity. Notably, we find that optimally we can use observed regions with seeing $< 1\farcs5$, and $r$-band extinction < 0.13 magnitudes, smaller than previously published results. Furthermore, we confirm that the uniformity of the SDSS photometry is minimally affected by the stripe geometry. We find that, overall, the two-point angular correlation function can be described by a power law, $\omega(\theta) = A_\omega \theta^{(1-\gamma)}$ with $\gamma \simeq 1.72$, over the range $0\fdg005$--$10\degr$. We also find similar relationships for the four magnitude subsamples, but the amplitude within the same angular interval for the four subsamples is found to decrease with fainter magnitudes, in agreement with previous results. We find that the systematic signals are well below the galaxy angular correlation function for angles less than approximately $5\degr$, which limits the modeling of galaxy angular correlations on larger scales. Finally, we present our custom, highly parallelized two-point correlation code that we used in this analysis.

Progress is being made in code discoverability and preservation, but as discussed at ADASS XXI, many codes still remain hidden from public view. With the Astrophysics Source Code Library (ASCL) now indexed by the SAO/NASA Astrophysics Data System (ADS), the introduction of a new journal, Astronomy & Computing, focused on astrophysics software, and the increasing success of education efforts such as Software Carpentry and SciCoder, the community has the opportunity to set a higher standard for its science by encouraging the release of software for examination and possible reuse. We assembled representatives of the community to present issues inhibiting code release and sought suggestions for tackling these factors. The session began with brief statements by panelists; the floor was then opened for discussion and ideas. Comments covered a diverse range of related topics and points of view, with apparent support for the propositions that algorithms should be readily available, code used to produce published scientific results should be made available, and there should be discovery mechanisms to allow these to be found easily. With increased use of resources such as GitHub (for code availability), ASCL (for code discovery), and a stated strong preference from the new journal Astronomy & Computing for code release, we expect to see additional progress over the next few years.

We use a quadratic estimator with KL-compression to calculate the angular power spectrum of a volume-limited Sloan Digital Sky Survey (SDSS) Data Release 7 (DR7) galaxy sample out to l = 200. We also determine the angular power spectrum of selected subsamples with photometric redshifts z < 0.3 and 0.3 < z < 0.4 to examine the possible evolution of the angular power spectrum, as well as early-type and late-type galaxy subsamples to examine the relative linear bias. In addition, we calculate the angular power spectrum of the SDSS DR7 main galaxy sample in a ~ 53.7 square degree area out to l = 1600 to determine the SDSS DR7 angular power spectrum to high multipoles. We perform a \chi^2 fit to compare the resulting angular power spectra to theoretical nonlinear angular power spectra to extract cosmological parameters and the linear bias. We find the best-fit cosmological parameters of \Omega_m = 0.267 +- 0.038 and \Omega_b = 0.045 +- 0.012. We find an overall linear bias of b = 1.075 +- 0.056, an early-type bias of b_e = 1.727 +- 0.065, and a late-type bias of b_l = 1.256 +- 0.051. Finally, we present evidence of a selective misclassification of late-type galaxies as stars by the SDSS photometric data reduction pipeline in areas of high stellar density (e.g., at low Galactic latitudes).

We calculate the angular power spectrum of galaxies selected from the Sloan Digital Sky Survey (SDSS) Data Release 7 (DR7) by using a quadratic estimation method with KL-compression. The primary data sample includes over 18 million galaxies covering more than 5,700 square degrees after masking areas with bright objects, reddening greater than 0.2 magnitudes, and seeing of more than 1.5 arcseconds. We test for systematic effects by calculating the angular power spectrum by SDSS stripe and find that these measurements are minimally affected by seeing and reddening. We calculate the angular power spectrum for l \leq 200 multipoles by using 40 bandpowers for the full sample, and l \leq 1000 multipoles using 50 bandpowers for individual stripes. We also calculate the angular power spectrum for this sample separated into 3 magnitude bins with mean redshifts of z = 0.171, z = 0.217, and z = 0.261 to examine the evolution of the angular power spectrum. We determine the theoretical linear angular power spectrum by projecting the 3D power spectrum to two dimensions for a basic comparison to our observational results. By minimizing the {\chi}^2 fit between these data and the theoretical linear angular power spectrum we measure a loosely-constrained fit of {\Omega}_m = 0.31^{+0.18}_{-0.11} with a linear bias of b = 0.94 \pm 0.04.

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.

We measure the angular auto-correlation functions (w) of SDSS galaxies selected to have photometric redshifts 0.1 < z < 0.4 and absolute r-band magnitudes Mr < -21.2. We split these galaxies into five overlapping redshift shells of width 0.1 and measure w in each subsample in order to investigate the evolution of SDSS galaxies. We find that the bias increases substantially with redshift - much more so than one would expect for a passively evolving sample. We use halo-model analysis to determine the best-fit halo-occupation-distribution (HOD) for each subsample, and the best-fit models allow us to interpret the change in bias physically. In order to properly interpret our best-fit HODs, we convert each halo mass to its z = 0 passively evolved bias (bo), enabling a direct comparison of the best-fit HODs at different redshifts. We find that the minimum halo bo required to host a galaxy decreases as the redshift decreases, suggesting that galaxies with Mr < -21.2 are forming in halos at the low-mass end of the HODs over our redshift range. We use the best-fit HODs to determine the change in occupation number divided by the change in mass of halos with constant bo and we find a sharp peak at bo ~ 0.9 - corresponding to an average halo mass of ~ 10^12Msol/h. We thus present the following scenario: the bias of galaxies with Mr < -21.2 decreases as the Universe evolves because these galaxies form in halos of mass ~ 10^12Msol/h (independent of redshift), and the bias of these halos naturally decreases as the Universe evolves.

A survey that can cover the sky in optical bands over wide fields to faint magnitudes with a fast cadence will enable many of the exciting science opportunities of the next decade. The Large Synoptic Survey Telescope (LSST) will have an effective aperture of 6.7 meters and an imaging camera with field of view of 9.6 deg^2, and will be devoted to a ten-year imaging survey over 20,000 deg^2 south of +15 deg. Each pointing will be imaged 2000 times with fifteen second exposures in six broad bands from 0.35 to 1.1 microns, to a total point-source depth of r~27.5. The LSST Science Book describes the basic parameters of the LSST hardware, software, and observing plans. The book discusses educational and outreach opportunities, then goes on to describe a broad range of science that LSST will revolutionize: mapping the inner and outer Solar System, stellar populations in the Milky Way and nearby galaxies, the structure of the Milky Way disk and halo and other objects in the Local Volume, transient and variable objects both at low and high redshift, and the properties of normal and active galaxies at low and high redshift. It then turns to far-field cosmological topics, exploring properties of supernovae to z~1, strong and weak lensing, the large-scale distribution of galaxies and baryon oscillations, and how these different probes may be combined to constrain cosmological models and the physics of dark energy.

We measure the angular 2-point correlation functions of galaxies in a volume limited, photometrically selected galaxy sample from the fifth data release of the Sloan Digital Sky Survey. We split the sample both by luminosity and galaxy type and use a halo-model analysis to find halo-occupation distributions that can simultaneously model the clustering of all, early-, and late-type galaxies in a given sample. Our results for the full galaxy sample are generally consistent with previous results using the SDSS spectroscopic sample, taking the differences between the median redshifts of the photometric and spectroscopic samples into account. We find that our early- and late- type measurements cannot be fit by a model that allows early- and late-type galaxies to be well-mixed within halos. Instead, we introduce a new model that segregates early- and late-type galaxies into separate halos to the maximum allowed extent. We determine that, in all cases, it provides a good fit to our data and thus provides a new statistical description of the manner in which early- and late-type galaxies occupy halos.

We analyze the cross-correlation of 2,705 unambiguously intervening Mg II (2796,2803A) quasar absorption line systems with 1,495,604 luminous red galaxies (LRGs) from the Fifth Data Release of the Sloan Digital Sky Survey within the redshift range 0.36<=z<=0.8. We confirm with high precision a previously reported weak anti-correlation of equivalent width and dark matter halo mass, measuring the average masses to be log M_h(M_[solar]h^-1)=11.29 [+0.36,-0.62] and log M_h(M_[solar]h^-1)=12.70 [+0.53,-1.16] for systems with W[2796A]>=1.4A and 0.8A<=W[2796A]<1.4A, respectively. Additionally, we investigate the significance of a number of potential sources of bias inherent in absorber-LRG cross-correlation measurements, including absorber velocity distributions and the weak lensing of background quasars, which we determine is capable of producing a 20-30% bias in angular cross-correlation measurements on scales less than 2'. We measure the Mg II - LRG cross-correlation for 719 absorption systems with v<60,000 km s^-1 in the quasar rest frame and find that these associated absorbers typically reside in dark matter haloes that are ~10-100 times more massive than those hosting unambiguously intervening Mg II absorbers. Furthermore, we find evidence for evolution of the redshift number density, dN/dz, with 2-sigma significance for the strongest (W>2.0A) absorbers in the DR5 sample. This width-dependent dN/dz evolution does not significantly affect the recovered equivalent width-halo mass anti-correlation and adds to existing evidence that the strongest Mg II absorption systems are correlated with an evolving population of field galaxies at z<0.8, while the non-evolving dN/dz of the weakest absorbers more closely resembles that of the LRG population.

We present measurements of the quasar two-point correlation function, \xi_{Q}, over the redshift range z=0.3-2.2 based upon data from the SDSS. Using a homogeneous sample of 30,239 quasars with spectroscopic redshifts from the DR5 Quasar Catalogue, our study represents the largest sample used for this type of investigation to date. With this redshift range and an areal coverage of approx 4,000 deg^2, we sample over 25 h^-3 Gpc^3 (comoving) assuming the current LCDM cosmology. Over this redshift range, we find that the redshift-space correlation function, xi(s), is adequately fit by a single power-law, with s_{0}=5.95+/-0.45 h^-1 Mpc and \gamma_{s}=1.16+0.11-0.16 when fit over s=1-25 h^-1 Mpc. Using the projected correlation function we calculate the real-space correlation length, r_{0}=5.45+0.35-0.45 h^-1 Mpc and \gamma=1.90+0.04-0.03, over scales of rp=1-130 h^-1 Mpc. Dividing the sample into redshift slices, we find very little, if any, evidence for the evolution of quasar clustering, with the redshift-space correlation length staying roughly constant at s_{0} ~ 6-7 h^-1 Mpc at z<2.2 (and only increasing at redshifts greater than this). Comparing our clustering measurements to those reported for X-ray selected AGN at z=0.5-1, we find reasonable agreement in some cases but significantly lower correlation lengths in others. We find that the linear bias evolves from b~1.4 at z=0.5 to b~3 at z=2.2, with b(z=1.27)=2.06+/-0.03 for the full sample. We compare our data to analytical models and infer that quasars inhabit dark matter haloes of constant mass M ~2 x 10^12 h^-1 M_Sol from redshifts z~2.5 (the peak of quasar activity) to z~0. [ABRIDGED]

We explore the multidimensional, multiwavelength selection of quasars from mid-IR (MIR) plus optical data, specifically from Spitzer-IRAC and the Sloan Digital Sky Survey (SDSS). We apply modern statistical techniques to combined Spitzer MIR and SDSS optical data, allowing up to 8-D color selection of quasars. Using a Bayesian selection method, we catalog 5546 quasar candidates to an 8.0 um depth of 56 uJy over an area of ~24 sq. deg; ~70% of these candidates are not identified by applying the same Bayesian algorithm to 4-color SDSS optical data alone. Our selection recovers 97.7% of known type 1 quasars in this area and greatly improves the effectiveness of identifying 3.5<z<5 quasars. Even using only the two shortest wavelength IRAC bandpasses, it is possible to use our Bayesian techniques to select quasars with 97% completeness and as little as 10% contamination. This sample has a photometric redshift accuracy of 93.6% (Delta Z +/-0.3), remaining roughly constant when the two reddest MIR bands are excluded. While our methods are designed to find type 1 (unobscured) quasars, as many as 1200 of the objects are type 2 (obscured) quasar candidates. Coupling deep optical imaging data with deep mid-IR data could enable selection of quasars in significant numbers past the peak of the quasar luminosity function (QLF) to at least z~4. Such a sample would constrain the shape of the QLF and enable quasar clustering studies over the largest range of redshift and luminosity to date, yielding significant gains in our understanding of quasars and the evolution of galaxies.

Using a homogenous sample of 38,208 quasars with a sky coverage of $4000 {\rm deg^2}$ drawn from the SDSS Data Release Five quasar catalog, we study the dependence of quasar clustering on luminosity, virial black hole mass, quasar color, and radio loudness. At $z<2.5$, quasar clustering depends weakly on luminosity and virial black hole mass, with typical uncertainty levels $\sim 10%$ for the measured correlation lengths. These weak dependences are consistent with models in which substantial scatter between quasar luminosity, virial black hole mass and the host dark matter halo mass has diluted any clustering difference, where halo mass is assumed to be the relevant quantity that best correlates with clustering strength. However, the most luminous and most massive quasars are more strongly clustered (at the $\sim 2\sigma$ level) than the remainder of the sample, which we attribute to the rapid increase of the bias factor at the high-mass end of host halos. We do not observe a strong dependence of clustering strength on quasar colors within our sample. On the other hand, radio-loud quasars are more strongly clustered than are radio-quiet quasars matched in redshift and optical luminosity (or virial black hole mass), consistent with local observations of radio galaxies and radio-loud type 2 AGN. Thus radio-loud quasars reside in more massive and denser environments in the biased halo clustering picture. Using the Sheth et al.(2001) formula for the linear halo bias, the estimated host halo mass for radio-loud quasars is $\sim 10^{13} h^{-1}M_\odot$, compared to $\sim 2\times 10^{12} h^{-1}M_\odot$ for radio-quiet quasar hosts at $z\sim 1.5$.

We present the final spectroscopic QSO catalogue from the 2dF-SDSS LRG and QSO (2SLAQ) Survey. This is a deep, 18<g<21.85 (extinction corrected), sample aimed at probing in detail the faint end of the broad line AGN luminosity distribution at z<2.6. The candidate QSOs were selected from SDSS photometry and observed spectroscopically with the 2dF spectrograph on the Anglo-Australian Telescope. This sample covers an area of 191.9 deg^2 and contains new spectra of 16326 objects, of which 8764 are QSOs, and 7623 are newly discovered (the remainder were previously identified by the 2QZ and SDSS surveys). The full QSO sample (including objects previously observed in the SDSS and 2QZ surveys) contains 12702 QSOs. The new 2SLAQ spectroscopic data set also contains 2343 Galactic stars, including 362 white dwarfs, and 2924 narrow emission line galaxies with a median redshift of z=0.22. We present detailed completeness estimates for the survey, based on modelling of QSO colours, including host galaxy contributions. This calculation shows that at g~21.85 QSO colours are significantly affected by the presence of a host galaxy up to redshift z~1 in the SDSS ugriz bands. In particular we see a significant reddening of the objects in g-i towards fainter g-band magnitudes. This reddening is consistent with the QSO host galaxies being dominated by a stellar population of age at least 2-3 Gyr. The full catalogue, including completeness estimates, is available on-line at http://www.2slaq.info/

We present a catalog of 1,172,157 quasar candidates selected from the photometric imaging data of the Sloan Digital Sky Survey (SDSS). The objects are all point sources to a limiting magnitude of i=21.3 from 8417 sq. deg. of imaging from SDSS Data Release 6 (DR6). This sample extends our previous catalog by using the latest SDSS public release data and probing both UV-excess and high-redshift quasars. While the addition of high-redshift candidates reduces the overall efficiency (quasars:quasar candidates) of the catalog to ~80%, it is expected to contain no fewer than 850,000 bona fide quasars -- ~8 times the number of our previous sample, and ~10 times the size of the largest spectroscopic quasar catalog. Cross-matching between our photometric catalog and spectroscopic quasar catalogs from both the SDSS and 2dF Surveys, yields 88,879 spectroscopically confirmed quasars. For judicious selection of the most robust UV-excess sources (~500,000 objects in all), the efficiency is nearly 97% -- more than sufficient for detailed statistical analyses. The catalog's completeness to type 1 (broad-line) quasars is expected to be no worse than 70%, with most missing objects occurring at z<0.7 and 2.5<z<3.0. In addition to classification information, we provide photometric redshift estimates (typically good to Delta z +/- 0.3 [2 sigma]) and cross-matching with radio, X-ray, and proper motion catalogs. Finally, we consider the catalog's utility for determining the optical luminosity function of quasars and are able to confirm the flattening of the bright-end slope of the quasar luminosity function at z~4 as compared to z~2.

We explore how the local environment is related to the redshift, type, and luminosity of active galactic nuclei (AGN). Recent simulations and observations are converging on the view that the extreme luminosity of quasars is fueled in major mergers of gas-rich galaxies. In such a picture, quasars are expected to be located in regions with a higher density of galaxies on small scales where mergers are more likely to take place. However, in this picture, the activity observed in low-luminosity AGN is due to secular processes that are less dependent on the local galaxy density. To test this hypothesis, we compare the local photometric galaxy density on kiloparsec scales around spectroscopic Type I and Type II quasars to the local density around lower luminosity spectroscopic Type I and Type II AGN. To minimize projection effects and evolution in the photometric galaxy sample we use to characterize AGN environments, we place our random control sample at the same redshift as our AGN and impose a narrow redshift window around both the AGN and control targets. We find that higher luminosity AGN have more overdense environments compared to lower luminosity AGN on all scales out to our $2\Mpchseventy$ limit. Additionally, in the range $0.3\leqslant z\leqslant 0.6$, Type II quasars have similarly overdense environments to those of bright Type I quasars on all scales out to our $2\Mpchseventy$ limit, while the environment of dimmer Type I quasars appears to be less overdense than the environment of Type II quasars. We see increased overdensity for Type II AGN compared to Type I AGN on scales out to our limit of $2\Mpchseventy$ in overlapping redshift ranges. We also detect marginal evidence for evolution in the number of galaxies within $2\Mpchseventy$ of a quasar with redshift.

This paper provides an evaluation of SGI RASCTM RC100 technology from a computational science software developer's perspective. A brute force implementation of a two-point angular correlation function is used as a test case application. The computational kernel of this test case algorithm is ported to the Mitrion-C programming language and compiled, targeting the RC100 hardware. We explore several code optimization techniques and report performance results for different designs. We conclude the paper with an analysis of this system based on our observations while implementing the test case. Overall, the hardware platform and software development tools were found to be satisfactory for accelerating computationally intensive applications, however, several system improvements are desirable.

We present a novel technique to measure $\sigma_8$, by measuring the dependence of the second-order bias of a density field on $\sigma_8$ using two separate techniques. Each technique employs area-averaged angular correlation functions ($\bar{\omega}_N$), one relying on the shape of $\bar{\omega}_2$, the other relying on the amplitude of $s_3$ ($s_3 =\bar{\omega}_3/\bar{\omega}_2^2$). We confirm the validity of the method by testing it on a mock catalog drawn from Millennium Simulation data and finding $\sigma_8^{measured}- \sigma_8^{true} = -0.002 \pm 0.062$. We create a catalog of photometrically selected LRGs from SDSS DR5 and separate it into three distinct data sets by photometric redshift, with median redshifts of 0.47, 0.53, and 0.61. Measurements of $c_2$, and $\sigma_8$ are made for each data set, assuming flat geometry and WMAP3 best-fit priors on $\Omega_m$, $h$, and $\Gamma$. We find, with increasing redshfit, $c_2 = 0.09 \pm 0.04$, $0.09 \pm 0.05$, and $0.09 \pm 0.03$ and $\sigma_8 = 0.78 \pm 0.08$, $0.80 \pm 0.09$, and $0.80 \pm 0.09$. We combine these three consistent $\sigma_8$ measurements to produce the result $\sigma_8 = 0.79 \pm 0.05$. Allowing the parameters $\Omega_m$, $h$, and $\Gamma$ to vary within their WMAP3 1$\sigma$ error, we find that the best-fit $\sigma_8$ does not change by more than 8% and we are thus confident our measurement is accurate to within 10%. We anticipate that future surveys, such as Pan-STARRS, DES, and LSST, will be able to employ this method to measure $\sigma_8$ to great precision, and will serve as an important check, complementary, on the values determined via more established methods.

We apply machine learning in the form of a nearest neighbor instance-based algorithm (NN) to generate full photometric redshift probability density functions (PDFs) for objects in the Fifth Data Release of the Sloan Digital Sky Survey (SDSS DR5). We use a conceptually simple but novel application of NN to generate the PDFs - perturbing the object colors by their measurement error - and using the resulting instances of nearest neighbor distributions to generate numerous individual redshifts. When the redshifts are compared to existing SDSS spectroscopic data, we find that the mean value of each PDF has a dispersion between the photometric and spectroscopic redshift consistent with other machine learning techniques, being sigma = 0.0207 +/- 0.0001 for main sample galaxies to r < 17.77 mag, sigma = 0.0243 +/- 0.0002 for luminous red galaxies to r < ~19.2 mag, and sigma = 0.343 +/- 0.005 for quasars to i < 20.3 mag. The PDFs allow the selection of subsets with improved statistics. For quasars, the improvement is dramatic: for those with a single peak in their probability distribution, the dispersion is reduced from 0.343 to sigma = 0.117 +/- 0.010, and the photometric redshift is within 0.3 of the spectroscopic redshift for 99.3 +/- 0.1% of the objects. Thus, for this optical quasar sample, we can virtually eliminate 'catastrophic' photometric redshift estimates. In addition to the SDSS sample, we incorporate ultraviolet photometry from the Third Data Release of the Galaxy Evolution Explorer All-Sky Imaging Survey (GALEX AIS GR3) to create PDFs for objects seen in both surveys. For quasars, the increased coverage of the observed frame UV of the SED results in significant improvement over the full SDSS sample, with sigma = 0.234 +/- 0.010. We demonstrate that this improvement is genuine. [Abridged]

We present recent results from the LCDM (Laboratory for Cosmological Data Mining; http://lcdm.astro.uiuc.edu) collaboration between UIUC Astronomy and NCSA to deploy supercomputing cluster resources and machine learning algorithms for the mining of terascale astronomical datasets. This is a novel application in the field of astronomy, because we are using such resources for data mining, and not just performing simulations. Via a modified implementation of the NCSA cyberenvironment Data-to-Knowledge, we are able to provide improved classifications for over 100 million stars and galaxies in the Sloan Digital Sky Survey, improved distance measures, and a full exploitation of the simple but powerful k-nearest neighbor algorithm. A driving principle of this work is that our methods should be extensible from current terascale datasets to upcoming petascale datasets and beyond. We discuss issues encountered to-date, and further issues for the transition to petascale. In particular, disk I/O will become a major limiting factor unless the necessary infrastructure is implemented.

Repeat scans by the Sloan Digital Sky Survey (SDSS) of a 278 square degree stripe along the Celestial equator have yielded an average of over 10 observations each for nearly 8,000 spectroscopically confirmed quasars. Over 2500 of these quasars are in the redshift range such that the CIV emission line is visible in the SDSS spectrum. Utilising the width of these CIV lines and the luminosity of the nearby continuum, we estimate black hole masses for these objects. In an effort to isolate the effects of black hole mass and luminosity on the photometric variability of our dataset, we create several subsamples by binning in these two physical parameters. By comparing the ensemble structure functions of the quasars in these bins, we are able to reproduce the well-known anticorrelation between luminosity and variability, now showing that this anticorrelation is independent of the black hole mass. In addition, we find a correlation between variability and the mass of the central black hole. By combining these two relations, we identify the Eddington ratio as a possible driver of quasar variability, most likely due to differences in accretion efficiency.

The objective of our research is to demonstrate the practical usage and orders of magnitude speedup of real-world applications by using alternative technologies to support high performance computing. Currently, the main barrier to the widespread adoption of this technology is the lack of development tools and case studies that typically impede non-specialists that might otherwise develop applications that could leverage these technologies. By partnering with the Innovative Systems Laboratory at the National Center for Supercomputing, we have obtained access to several novel technologies, including several Field-Programmable Gate Array (FPGA) systems, NVidia Graphics Processing Units (GPUs), and the STI Cell BE platform. Our goal is to not only demonstrate the capabilities of these systems, but to also serve as guides for others to follow in our path. To date, we have explored the efficacy of the SRC-6 MAP-C and MAP-E and SGI RASC Athena and RC100 reconfigurable computing platforms in supporting a two-point correlation function which is used in a number of different scientific domains. In a brute force test, the FPGA based single-processor system has achieved an almost two orders of magnitude speedup over a single-processor CPU system. We are now developing implementations of this algorithm on other platforms, including one using a GPU. Given the considerable efforts of the cosmology community in optimizing these classes of algorithms, we are currently working to implement an optimized version of the basic family of correlation functions by using tree-based data structures. Finally, we are also exploring other algorithms, such as instance-based classifiers, power spectrum estimators, and higher-order correlation functions that are also commonly used in a wide range of scientific disciplines.

Angular power spectra are an important measure of the angular clustering of a given distribution. In Cosmology, they are applied to such vastly different observations as galaxy surveys that cover a fraction of the sky and the Cosmic Microwave Background that covers the entire sky, to obtain fundamental parameters that determine the structure and evolution of the universe. The calculation of an angular power spectrum, however, is complex and the optimization of these calculations is a necessary consideration for current and forthcoming observational surveys. In this work, we present preliminary results of implementing angular power spectrum estimation scheme on a high-performance reconfigurable computing platform.

In this case study, we investigate the impact of workload balance on the performance of multi-FPGA codes. We start with an application in which two distinct kernels run in parallel on two SRC-6 MAP processors. We observe that one of the MAP processors is idle 18% of the time while the other processor is fully utilized. We investigate a task redistribution schema which serializes the execution of the two kernels, yet parallelizes execution of each individual kernel by spreading the workload between two MAP processors. This implementation results in a near 100% utilization of both MAP processors and the overall application performance is improved by 9%.

We present cosmological results from the statistics of lensed quasars in the Sloan Digital Sky Survey (SDSS) Quasar Lens Search. By taking proper account of the selection function, we compute the expected number of quasars lensed by early-type galaxies and their image separation distribution assuming a flat universe, which is then compared with 7 lenses found in the SDSS Data Release 3 to derive constraints on dark energy under strictly controlled criteria. For a cosmological constant model (w=-1) we obtain \Omega_\Lambda=0.74^{+0.11}_{-0.15}(stat.)^{+0.13}_{-0.06}(syst.). Allowing w to be a free parameter we find \Omega_M=0.26^{+0.07}_{-0.06}(stat.)^{+0.03}_{-0.05}(syst.) and w=-1.1\pm0.6(stat.)^{+0.3}_{-0.5}(syst.) when combined with the constraint from the measurement of baryon acoustic oscillations in the SDSS luminous red galaxy sample. Our results are in good agreement with earlier lensing constraints obtained using radio lenses, and provide additional confirmation of the presence of dark energy consistent with a cosmological constant, derived independently of type Ia supernovae.

We report the first results of our systematic search for strongly lensed quasars using the spectroscopically confirmed quasars in the Sloan Digital Sky Survey (SDSS). Among 46,420 quasars from the SDSS Data Release 3 (~4188 deg^2), we select a subsample of 22,683 quasars that are located at redshifts between 0.6 and 2.2 and are brighter than the Galactic extinction corrected i-band magnitude of 19.1. We identify 220 lens candidates from the quasar subsample, for which we conduct extensive and systematic follow-up observations in optical and near-infrared wavebands, in order to construct a complete lensed quasar sample at image separations between 1'' and 20'' and flux ratios of faint to bright lensed images larger than 10^{-0.5}. We construct a statistical sample of 11 lensed quasars. Ten of these are galaxy-scale lenses with small image separations (~1''-2'') and one is a large separation (15'') system which is produced by a massive cluster of galaxies, representing the first statistical sample of lensed quasars including both galaxy- and cluster-scale lenses. The Data Release 3 spectroscopic quasars contain an additional 11 lensed quasars outside the statistical sample.

We use the Fourth Data Release of the Sloan Digital Sky Survey to investigate the relation between galaxy rest frame u-r colour, morphology, as described by the concentration and Sersic indices, and environmental density, for a sample of 79,553 galaxies at z < ~0.1. We split the samples according to density and luminosity and recover the expected bimodal distribution in the colour-morphology plane, shown especially clearly by this subsampling. We quantify the bimodality by a sum of two Gaussians on the colour and morphology axes and show that, for the red/early-type population both colour and morphology do not change significantly as a function of density. For the blue/late-type population, with increasing density the colour becomes redder but the morphology again does not change significantly. Both populations become monotonically redder and of earlier type with increasing luminosity. There is no significant qualitative difference between the behaviour of the two morphological measures. We supplement the morphological sample with 13,655 galaxies assigned Hubble types by an artificial neural network. We find, however, that the resulting distribution is less well described by two Gaussians. Therefore, there are either more than two significant morphological populations, physical processes not seen in colour space, or the Hubble type, particularly the different subtypes of spirals Sa-Sd, has an irreducible fuzziness when related to environmental density. For each of the three measures of morphology, on removing the density relation due to it, we recover a strong residual relation in colour. However, on similarly removing the colour-density relation there is no evidence for a residual relation due to morphology. [Abridged]

We present recent results from the Laboratory for Cosmological Data Mining (http://lcdm.astro.uiuc.edu) at the National Center for Supercomputing Applications (NCSA) to provide robust classifications and photometric redshifts for objects in the terascale-class Sloan Digital Sky Survey (SDSS). Through a combination of machine learning in the form of decision trees, k-nearest neighbor, and genetic algorithms, the use of supercomputing resources at NCSA, and the cyberenvironment Data-to-Knowledge, we are able to provide improved classifications for over 100 million objects in the SDSS, improved photometric redshifts, and a full exploitation of the powerful k-nearest neighbor algorithm. This work is the first to apply the full power of these algorithms to contemporary terascale astronomical datasets, and the improvement over existing results is demonstrable. We discuss issues that we have encountered in dealing with data on the terascale, and possible solutions that can be implemented to deal with upcoming petascale datasets.

We present spectroscopy of binary quasar candidates selected from Data Release 4 of the Sloan Digital Sky Survey (SDSS DR4) using Kernel Density Estimation (KDE). We present 27 new sets of observations, 10 of which are binary quasars, roughly doubling the number of known $g < 21$ binaries with component separations of 3 to 6". Only 3 of 49 spectroscopically identified objects are non-quasars, confirming that the quasar selection efficiency of the KDE technique is $\sim95$%. Several of our observed binaries are wide-separation lens candidates that merit additional higher-resolution observations. One interesting pair may be an M star binary, or an M star-binary quasar superposition. Our candidates are initially selected by UV-excess ($u-g < 1$), but are otherwise selected irrespective of the relative colors of the quasar pair, and we thus use them to suggest optimal color similarity and photometric redshift approaches for targeting binary quasars, or projected quasar pairs. From a sample that is complete on proper scales of $23.7 < R_{prop} < 29.7\kpch$, we determine the projected quasar correlation function to be $W_p=24.0 \pm^{16.9}_{10.8}$, which is $2\sigma$ lower than recent estimates. We argue that our low $W_p$ estimates may indicate redshift evolution in the quasar correlation function from $z\sim1.9$ to $z\sim1.4$ on scales of $R_{prop} \sim25\kpch$. The size of this evolution broadly tracks quasar clustering on larger scales, consistent with merger-driven models of quasar origin. Although our sample alone is insufficient to detect evolution in quasar clustering on small scales, an $i$-selected DR6 KDE quasar catalog, which will contain several hundred $z \leqsim 5$ binary quasars, could easily constrain any clustering evolution at $R_{prop} \sim25\kpch$.

We investigate the time-dependent variations of ultraviolet (UV) black hole mass estimates of quasars in the Sloan Digital Sky Survey (SDSS). From SDSS spectra of 615 high-redshift (1.69 < z < 4.75) quasars with spectra from two epochs, we estimate black hole masses, using a single-epoch technique which employs an additional, automated night-sky-line removal, and relies on UV continuum luminosity and CIV (1549A) emission line dispersion. Mass estimates show variations between epochs at about the 30% level for the sample as a whole. We determine that, for our full sample, measurement error in the line dispersion likely plays a larger role than the inherent variability, in terms of contributing to variations in mass estimates between epochs. However, we use the variations in quasars with r-band spectral signal-to-noise ratio greater than 15 to estimate that the contribution to these variations from inherent variability is roughly 20%. We conclude that these differences in black hole mass estimates between epochs indicate variability is not a large contributer to the current factor of two scatter between mass estimates derived from low- and high-ionization emission lines.

We present estimates of the N-point galaxy, area-averaged, angular correlation functions $\bar{\omega}_{N}$($\theta$) for $N$ = 2,...,7 for galaxies from the fifth data release of the Sloan Digital Sky Survey. Our parent sample is selected from galaxies with $18 \leq r < 21$, and is the largest ever used to study higher-order correlations. We subdivide this parent sample into two volume limited samples using photometric redshifts, and these two samples are further subdivided by magnitude, redshift, and color (producing early- and late-type galaxy samples) to determine the dependence of $\bar{\omega}_{N}$($\theta$) on luminosity, redshift, and galaxy-type. We measure $\bar{\omega}_{N}$($\theta$) using oversampling techniques and use them to calculate the projected, $s_{N}$. Using models derived from theoretical power-spectra and perturbation theory, we measure the bias parameters $b_1$ and $c_2$, finding that the large differences in both bias parameters ($b_1$ and $c_2$) between early- and late-type galaxies are robust against changes in redshift, luminosity, and $\sigma_8$, and that both terms are consistently smaller for late-type galaxies. By directly comparing their higher-order correlation measurements, we find large differences in the clustering of late-type galaxies at redshifts lower than 0.3 and those at redshifts higher than 0.3, both at large scales ($c_2$ is larger by $\sim0.5$ at $z > 0.3$) and small scales (large amplitudes are measured at small scales only for $z > 0.3$, suggesting much more merger driven star formation at $z > 0.3$). Finally, our measurements of $c_2$ suggest both that $\sigma_8 < 0.8$ and $c_2$ is negative.

We present the fourth edition of the Sloan Digital Sky Survey (SDSS) Quasar Catalog. The catalog contains 77,429 objects; this is an increase of over 30,000 entries since the previous edition. The catalog consists of the objects in the SDSS Fifth Data Release that have luminosities larger than M_i = -22.0 (in a cosmology with H_0 = 70 km/s/Mpc, Omega_M = 0.3, and Omega_Lambda = 0.7) have at least one emission line with FWHM larger than 1000 km/s, or have interesting/complex absorption features, are fainter than i=15.0, and have highly reliable redshifts. The area covered by the catalog is 5740 sq. deg. The quasar redshifts range from 0.08 to 5.41, with a median value of 1.48; the catalog includes 891 quasars at redshifts greater than four, of which 36 are at redshifts greater than five. Approximately half of the catalog quasars have i < 19; nearly all have i < 21. For each object the catalog presents positions accurate to better than 0.2 arcsec. rms per coordinate, five-band (ugriz) CCD-based photometry with typical accuracy of 0.03 mag, and information on the morphology and selection method. The catalog also contains basic radio, near-infrared, and X-ray emission properties of the quasars, when available, from other large-area surveys. The calibrated digital spectra cover the wavelength region 3800--9200A at a spectral resolution of ~2000. The spectra can be retrieved from the public database using the information provided in the catalog. The average SDSS colors of quasars as a function of redshift, derived from the catalog entries, are presented in tabular form. Approximately 96% of the objects in the catalog were discovered by the SDSS.

We apply instance-based machine learning in the form of a k-nearest neighbor algorithm to the task of estimating photometric redshifts for 55,746 objects spectroscopically classified as quasars in the Fifth Data Release of the Sloan Digital Sky Survey. We compare the results obtained to those from an empirical color-redshift relation (CZR). In contrast to previously published results using CZRs, we find that the instance-based photometric redshifts are assigned with no regions of catastrophic failure. Remaining outliers are simply scattered about the ideal relation, in a similar manner to the pattern seen in the optical for normal galaxies at redshifts z < ~1. The instance-based algorithm is trained on a representative sample of the data and pseudo-blind-tested on the remaining unseen data. The variance between the photometric and spectroscopic redshifts is sigma^2 = 0.123 +/- 0.002 (compared to sigma^2 = 0.265 +/- 0.006 for the CZR), and 54.9 +/- 0.7%, 73.3 +/- 0.6%, and 80.7 +/- 0.3% of the objects are within delta z < 0.1, 0.2, and 0.3 respectively. We also match our sample to the Second Data Release of the Galaxy Evolution Explorer legacy data and the resulting 7,642 objects show a further improvement, giving a variance of sigma^2 = 0.054 +/- 0.005, and 70.8 +/- 1.2%, 85.8 +/- 1.0%, and 90.8 +/- 0.7% of objects within delta z < 0.1, 0.2, and 0.3. We show that the improvement is indeed due to the extra information provided by GALEX, by training on the same dataset using purely SDSS photometry, which has a variance of sigma^2 = 0.090 +/- 0.007. Each set of results represents a realistic standard for application to further datasets for which the spectra are representative.

We present a time-variability analysis of 29 broad absorption line quasars (BALQSOs) observed in two epochs by the Sloan Digital Sky Survey (SDSS). These spectra are selected from a larger sample of BALQSOs with multiple observations by virtue of exhibiting a broad CIV $\lambda$1549 absorption trough separated from the rest frame of the associated emission peak by more than 3600 km s$^{-1}$. Detached troughs facilitate higher precision variability measurements, since the measurement of the absorption in these objects is not complicated by variation in the emission line flux. We have undertaken a statistical analysis of these detached-trough BALQSO spectra to explore the relationships between BAL features that are seen to vary and the dynamics of emission from the quasar central engine. We have measured variability within our sample, which includes three strongly variable BALs. We have also verified that the statistical behavior of the overall sample agrees with current model predictions and previous studies of BAL variability. Specifically, we observe that the strongest BAL variability occurs among the smallest equivalent width features and at velocities exceeding 12,000 km s$^{-1}$, as predicted by recent disk-wind modeling.

We combine the QSO samples from the 2dF QSO Redshift Survey (2QZ) and the 2dF-SDSS LRG and QSO Survey (2SLAQ) in order to investigate the clustering of z~1.4 QSOs and measure the correlation function. The clustering signal in z-space, projected along the sky direction, is similar to that previously obtained from 2QZ alone. By fitting the z-space correlation function and lifting the degeneracy between beta and Omega_m_0 by using linear theory predictions, we obtain beta(z=1.4) = 0.60+-0.12 and Omega_m_0=0.25+-0.08, implying a value for the QSO bias, b(z=1.4)=1.5+-0.2. We further find that QSO clustering does not depend strongly on luminosity at fixed redshift. This result is inconsistent with the expectation of simple `high peaks' biasing models where more luminous, rare QSOs are assumed to inhabit higher mass haloes. The data are more consistent with models which predict that QSOs of different luminosities reside in haloes of similar mass. We find that halo mass does not evolve strongly with redshift nor depend on QSO luminosity. We finally investigate how black hole mass correlates with luminosity and redshift and ascertain the relation between Eddington efficiency and black hole mass. Our results suggest that QSOs of different luminosities may contain black holes of similar mass.

Using ~300,000 photometrically classified quasars, by far the largest quasar sample ever used for such analyses, we study the redshift and luminosity evolution of quasar clustering on scales of ~50 kpc/h to ~20 Mpc/h from redshifts of z~0.75 to z~2.28. We parameterize our clustering amplitudes using realistic dark matter models, and find that a LCDM power spectrum provides a superb fit to our data with a redshift-averaged quasar bias of b_Q = 2.41+/-0.08 ($P_{<\chi^2}=0.847$) for $\sigma_8=0.9$. This represents a better fit than the best-fit power-law model ($\omega = 0.0493\pm0.0064\theta^ {-0.928\pm0.055}$; $P_{<\chi^2}=0.482$). We find b_Q increases with redshift. This evolution is significant at >99.6% using our data set alone, increasing to >99.9999% if stellar contamination is not explicitly parameterized. We measure the quasar classification efficiency across our full sample as a = 95.6 +/- ^{4.4}_{1.9}%, a star-quasar separation comparable with the star-galaxy separation in many photometric studies of galaxy clustering. We derive the mean mass of the dark matter halos hosting quasars as MDMH=(5.2+/-0.6)x10^{12} M_solar/h. At z~1.9 we find a $1.5\sigma$ deviation from luminosity-independent quasar clustering; this suggests that increasing our sample size by a factor of 1.8 could begin to constrain any luminosity dependence in quasar bias at z~2. Our results agree with recent studies of quasar environments at z < 0.4, which detected little luminosity dependence to quasar clustering on proper scales >50 kpc/h. At z < 1.6, our analysis suggests that b_Q is constant with luminosity to within ~0.6, and that, for g < 21, angular quasar autocorrelation measurements are unlikely to have sufficient statistical power at z < 1.6 to detect any luminosity dependence in quasars' clustering.

We study quasar clustering on small scales, modeling clustering amplitudes using halo-driven dark matter descriptions. From 91 pairs on scales <35 kpc/h, we detect only a slight excess in quasar clustering over our best-fit large-scale model. Integrated across all redshifts, the implied quasar bias is b_Q = 4.21+/-0.98 (b_Q = 3.93+/-0.71) at ~18 kpc/h (~28 kpc/h). Our best-fit (real-space) power index is ~-2 (i.e., $\xi(r) \propto r^{-2}$), implying steeper halo profiles than currently found in simulations. Alternatively, quasar binaries with separation <35 kpc/h may trace merging galaxies, with typical dynamical merger times t_d~(610+/-260)m^{-1/2} Myr/h, for quasars of host halo mass m x 10^{12} Msolar/h. We find UVX quasars at ~28 kpc/h cluster >5 times higher at z > 2, than at z < 2, at the $2.0\sigma$ level. However, as the space density of quasars declines as z increases, an excess of quasar binaries (over expectation) at z > 2 could be consistent with reduced merger rates at z > 2 for the galaxies forming UVX quasars. Comparing our clustering at ~28 kpc/h to a $\xi(r)=(r/4.8\Mpch)^{-1.53}$ power-law, we find an upper limit on any excess of a factor of 4.3+/-1.3, which, noting some caveats, differs from large excesses recently measured for binary quasars, at $2.2\sigma$. We speculate that binary quasar surveys that are biased to z > 2 may find inflated clustering excesses when compared to models fit at z < 2. We provide details of 111 photometrically classified quasar pairs with separations <0.1'. Spectroscopy of these pairs could significantly constrain quasar dynamics in merging galaxies.

We present evidence of a large angle correlation between the cosmic microwave background measured by WMAP and a catalog of photometrically detected quasars from the SDSS. The observed cross correlation is (0.30 +- 0.14) microK at zero lag, with a shape consistent with that expected for correlations arising from the integrated Sachs-Wolfe effect. The photometric redshifts of the quasars are centered at z ~ 1.5, making this the deepest survey in which such a correlation has been observed. Assuming this correlation is due to the ISW effect, this constitutes the earliest evidence yet for dark energy and it can be used to constrain exotic dark energy models.

Bivariate luminosity functions (LFs) are computed for galaxies in the New York Value-Added Galaxy Catalogue, based on the Sloan Digital Sky Survey Data Release 4. The galaxy properties investigated are the morphological type, inverse concentration index, Sersic index, absolute effective surface brightness, reference frame colours, absolute radius, eClass spectral type, stellar mass and galaxy environment. The morphological sample is flux-limited to galaxies with r < 15.9 and consists of 37,047 classifications to an RMS accuracy of +/- half a class in the sequence E, S0, Sa, Sb, Sc, Sd, Im. These were assigned by an artificial neural network, based on a training set of 645 eyeball classifications. The other samples use r < 17.77 with a median redshift of z ~ 0.08, and a limiting redshift of z < 0.15 to minimize the effects of evolution. Other cuts, for example in axis ratio, are made to minimize biases. A wealth of detail is seen, with clear variations between the LFs according to absolute magnitude and the second parameter. They are consistent with an early type, bright, concentrated, red population and a late type, faint, less concentrated, blue, star forming population. This bimodality suggests two major underlying physical processes, which in agreement with previous authors we hypothesize to be merger and accretion, associated with the properties of bulges and discs respectively. The bivariate luminosity-surface brightness distribution is fit with the Choloniewski function (a Schechter function in absolute magnitude and Gaussian in surface brightness). The fit is found to be poor, as might be expected if there are two underlying processes.

We present a comparison of optical and X-ray properties of galaxy clusters in the northern sky. We determine the recovery rate of X-ray detected clusters in the optical as a function of richness, redshift and X-ray luminosity, showing that the missed clusters are typically low contrast systems when observed optically. We employ four different statistical tests to test for the presence of substructure using optical two-dimensional data, finding that approximately 35% of the clusters show strong signs of substructure. However, the results are test-dependent, with variations also due to the magnitude range and radius utilized.We have also performed a comparison of X-ray luminosity and temperature with optical galaxy counts (richness). We find that the slope and scatter of the relations between richness and the X-ray properties are heavily dependent on the density contrast of the clusters. The selection of substructure-free systems does not improve the correlation between X-ray luminosity and richness, but this comparison also shows much larger scatter than one obtained using the X-ray temperature. In the latter case, the sample is significantly reduced because temperature measurements are available only for the most massive (and thus high contrast) systems. However, the comparison between temperature and richness is very sensitive to the exclusion of clusters showing signs of substructure. The correlation of X-ray luminosity and richness is based on the largest sample to date ($\sim$ 750 clusters), while tests involving temperature use a similar number of objects as previous works ($\lsim$100). The results presented here are in good agreement with existing literature.

We present new measurements of the luminosity function (LF) of Luminous Red Galaxies (LRGs) from the Sloan Digital Sky Survey (SDSS) and the 2dF-SDSS LRG and Quasar (2SLAQ) survey. We have carefully quantified, and corrected for, uncertainties in the K and evolutionary corrections, differences in the colour selection methods, and the effects of photometric errors, thus ensuring we are studying the same galaxy population in both surveys. Using a limited subset of 6326 SDSS LRGs (with 0.17<z<0.24) and 1725 2SLAQ LRGs (with 0.5 <z<0.6), for which the matching colour selection is most reliable, we find no evidence for any additional evolution in the LRG LF, over this redshift range, beyond that expected from a simple passive evolution model. This lack of additional evolution is quantified using the comoving luminosity density of SDSS and 2SLAQ LRGs, brighter than M_r - 5logh = -22.5, which are 2.51+/-0.03 x 10^-7 L_sun Mpc^-3 and 2.44+/-0.15 x 10^-7 L_sun Mpc^-3 respectively (<10% uncertainty). We compare our LFs to the COMBO-17 data and find excellent agreement over the same redshift range. Together, these surveys show no evidence for additional evolution (beyond passive) in the LF of LRGs brighter than M_r - 5logh = -21 (or brighter than L*). We test our SDSS and 2SLAQ LFs against a simple ``dry merger'' model for the evolution of massive red galaxies and find that at least half of the LRGs at z=0.2 must already have been well-assembled (with more than half their stellar mass) by z=0.6. This limit is barely consistent with recent results from semi-analytical models of galaxy evolution.

We present a spectroscopic survey of almost 15,000 candidate intermediate-redshift Luminous Red Galaxies (LRGs) brighter than i=19.8, observed with 2dF on the Anglo-Australian Telescope. The targets were selected photometrically from the Sloan Digital Sky Survey (SDSS) and lie along two narrow equatorial strips covering 180 sq deg. Reliable redshifts were obtained for 92% of the targets and the selection is very efficient: over 90% have redshifts between 0.45 and 0.8. More than 80% of the ~11,000 red galaxies have pure absorption-line spectra consistent with a passively-evolving old stellar population. The redshift, photometric and spatial distributions of the LRGs are described. The 2SLAQ data will be released publicly from mid-2006, providing a powerful resource for observational cosmology and the study of galaxy evolution.

We provide classifications for all 143 million non-repeat photometric objects in the Third Data Release of the Sloan Digital Sky Survey (SDSS) using decision trees trained on 477,068 objects with SDSS spectroscopic data. We demonstrate that these star/galaxy classifications are expected to be reliable for approximately 22 million objects with r < ~20. The general machine learning environment Data-to-Knowledge and supercomputing resources enabled extensive investigation of the decision tree parameter space. This work presents the first public release of objects classified in this way for an entire SDSS data release. The objects are classified as either galaxy, star or nsng (neither star nor galaxy), with an associated probability for each class. To demonstrate how to effectively make use of these classifications, we perform several important tests. First, we detail selection criteria within the probability space defined by the three classes to extract samples of stars and galaxies to a given completeness and efficiency. Second, we investigate the efficacy of the classifications and the effect of extrapolating from the spectroscopic regime by performing blind tests on objects in the SDSS, 2dF Galaxy Redshift and 2dF QSO Redshift (2QZ) surveys. Given the photometric limits of our spectroscopic training data, we effectively begin to extrapolate past our star-galaxy training set at r ~ 18. By comparing the number counts of our training sample with the classified sources, however, we find that our efficiencies appear to remain robust to r ~ 20. As a result, we expect our classifications to be accurate for 900,000 galaxies and 6.7 million stars, and remain robust via extrapolation for a total of 8.0 million galaxies and 13.9 million stars. [Abridged]

We present estimates of the N-point galaxy area-averaged angular correlation functions wN for N = 2,...,7 from the third data release of the Sloan Digital Sky Survey (SDSS). The sample was selected from galaxies with 18 < r < 21, and is the largest ever used to study higher-order correlations. The measured wN are used to calculate the projected, sN, and real space, SN, hierarchical amplitudes. This produces highly-precise measurements over 0.2 to 10 h-1 Mpc, which are consistent with Gaussian primordial density fluctuations. The measurements suggest that higher-order galaxy bias is non-negligible, as defining b1 = 1 yields c2 = -0.24 +/- 0.08. We report the first SDSS measurement of marginally significant third-order bias, c3 = 0.98 +/- 0.89, which suggests that bias terms may be significant to even higher order. Previous measurements of c2 have yielded inconsistent results. Inconsistencies would be expected if different data sets sample different galaxy types, especially if different galaxy types exhibit different higher-order bias. We find early-type galaxies exhibit significantly different behavior than late-types at both small and large scales. At large scales (r > 1 h-1 Mpc), we find the SN for late-type galaxies are lower than for early-types, implying a significant difference between their higher-order bias. We find b1,early = 1.36 +/- 0.04, c2,early = 0.30 +/- 0.10, b1,late = 0.81 +/- 0.03, and c2,late = -0.70 +/- 0.08. Our results are robust against the systematic effects of reddening and seeing. The latter introduces minor structure in wN.

With close pairs of quasars at different redshifts, a background quasar sightline can be used to study a foreground quasar's environment in absorption. We search 149 moderate resolution background quasar spectra, from Gemini, Keck, the MMT, and the SDSS to survey Lyman Limit Systems (LLSs) and Damped Ly-alpha systems (DLAs) in the vicinity of 1.8 < z < 4.0 luminous foreground quasars. A sample of 27 new quasar-absorber pairs is uncovered with column densities, 17.2 < log (N_HI/cm^2) < 20.9, and transverse (proper) distances of 22 kpc/h < R < 1.7 Mpc/h, from the foreground quasars. If they emit isotropically, the implied ionizing photon fluxes are a factor of ~ 5-8000 times larger than the ambient extragalactic UV background over this range of distances. The observed probability of intercepting an absorber is very high for small separations: six out of eight projected sightlines with transverse separations R < 150 kpc/h have an absorber coincident with the foreground quasar, of which four have log N_HI > 10^19. The covering factor of log N_HI > 10^19 absorbers is thus ~ 50 % (4/8) on these small scales, whereas < 2% would have been expected at random. There are many cosmological applications of these new sightlines: they provide laboratories for studying fluorescent Ly-alpha recombination radiation from LLSs, constrain the environments, emission geometry, and radiative histories of quasars, and shed light on the physical nature of LLSs and DLAs.

We determine the number counts and z=0-5 luminosity function for a well-defined, homogeneous sample of quasars from the Sloan Digital Sky Survey (SDSS). We conservatively define the most uniform statistical sample possible, consisting of 15,343 quasars within an effective area of 1622 deg^2 that was derived from a parent sample of 46,420 spectroscopically confirmed broad-line quasars in the 5282 deg^2 of imaging data from SDSS Data Release Three. The sample extends from i=15 to i=19.1 at z<3 and to i=20.2 for z>3. The number counts and luminosity function agree well with the results of the 2dF QSO Survey, but the SDSS data probe to much higher redshifts than does the 2dF sample. The number density of luminous quasars peaks between redshifts 2 and 3, although uncertainties in the selection function in this range do not allow us to determine the peak redshift more precisely. Our best fit model has a flatter bright end slope at high redshift than at low redshift. For z<2.4 the data are best fit by a redshift-independent slope of beta = -3.1 (Phi(L) propto L^beta). Above z=2.4 the slope flattens with redshift to beta=-2.37 at z=5. This slope change, which is significant at a >5-sigma level, must be accounted for in models of the evolution of accretion onto supermassive black holes.

We present variability and multi-wavelength photometric information for the 933 known quasars in the QUEST Variability Survey. These quasars are grouped into variable and non-variable populations based on measured variability confidence levels. In a time-limited synoptic survey, we detect an anti-correlation between redshift and the likelihood of variability. Our comparison of variability likelihood to radio, IR, and X-ray data is consistent with earlier quasar studies. Using already-known quasars as a template, we introduce a light curve morphology algorithm that provides an efficient method for discriminating variable quasars from periodic variable objects in the absence of spectroscopic information. The establishment of statistically robust trends and efficient, non-spectroscopic selection algorithms will aid in quasar identification and categorization in upcoming massive synoptic surveys. Finally, we report on three interesting variable quasars, including variability confirmation of the BL Lac candidate PKS 1222+037.

We examine the variability of the high-ionizaton C IV line in a sample of 105 quasars observed at multiple epochs by the Sloan Digital Sky Survey. We find a strong correlation between the change in the C IV line flux and the change in the line width, but no correlations between the change in flux and changes in line center and skewness. The relation between line flux change and line width change is consistent with a model in which a broad line base varies with greater amplitude than the line core. The objects studied here are more luminous and at higher redshift than those normally studied for variability, ranging in redshift from 1.65 to 4.00 and in absolute r-band magnitude from roughly -24 to -28. Using moment analysis line-fitting techniques, we measure line fluxes, centers, widths and skewnesses for the C IV line at two epochs for each object. The well-known Baldwin Effect is seen for these objects, with a slope beta = -0.22. The sample has a median intrinsic Baldwin Effect slope of beta = -0.85; the C IV lines in these high-luminosity quasars appear to be less responsive to continuum variations than those in lower luminosity AGN. Additionally, we find no evidence for variability of the well known blueshift of the C IV line with respect to the low-ionization Mg II line in the highest flux objects, indicating that this blueshift might be useful as a measure of orientation.

We present new measurements of the quasar autocorrelation from a sample of \~80,000 photometrically-classified quasars taken from SDSS DR1. We find a best-fit model of $\omega(\theta) = (0.066\pm^{0.026}_{0.024})\theta^{-(0.98\pm0.15)}$ for the angular autocorrelation, consistent with estimates from spectroscopic quasar surveys. We show that only models with little or no evolution in the clustering of quasars in comoving coordinates since z~1.4 can recover a scale-length consistent with local galaxies and Active Galactic Nuclei (AGNs). A model with little evolution of quasar clustering in comoving coordinates is best explained in the current cosmological paradigm by rapid evolution in quasar bias. We show that quasar biasing must have changed from b_Q~3 at a (photometric) redshift of z=2.2 to b_Q~1.2-1.3 by z=0.75. Such a rapid increase with redshift in biasing implies that quasars at z~2 cannot be the progenitors of modern L* objects, rather they must now reside in dense environments, such as clusters. Similarly, the duration of the UVX quasar phase must be short enough to explain why local UVX quasars reside in essentially unbiased structures. Our estimates of b_Q are in good agreement with recent spectroscopic results, which demonstrate the implied evolution in b_Q is consistent with quasars inhabiting halos of similar mass at every redshift. Treating quasar clustering as a function of both redshift and luminosity, we find no evidence for luminosity dependence in quasar clustering, and that redshift evolution thus affects quasar clustering more than changes in quasars' luminosity. We provide a new method for quantifying stellar contamination in photometrically-classified quasar catalogs via the correlation function.

Advanced instruments in a variety of scientific domains are collecting massive amounts of data that must be post-processed and organized to support scientific research activities. Astronomers have been pioneers in the use of databases to host highly structured repositories of sky survey data. As more powerful telescopes come online, the increased volume and complexity of the data collected poses enormous challenges to state-of-the-art database systems and data-loading techniques. When the data source is an instrument taking ongoing samples, the database loading must, at a minimum, keep up with the data-acquisition rate. These challenges are being faced not only by the astronomy community, but also by other scientific disciplines interested in building scalable databases to house multi-terabyte archives of complex structured data. In this paper we present SkyLoader, our novel framework for fast and scalable data loading that is being used to populate a multi-table, multi-terabyte database repository for the Palomar-Quest sky survey. Our framework consists of an efficient algorithm for bulk loading, an effective data structure to support data integrity and proper error handling during the loading process, support for optimized parallelism that matches the number of concurrent loaders with the database host capabilities, and guidelines for database and system tuning. Performance studies showing the positive effects of the adopted strategies are also presented. Our parallel bulk loading with array buffering technique has made fast population of a multi-terabyte repository a reality, reducing the loading time for a 40-gigabyte data set from more than 20 hours to less than 3 hours. We believe our framework offers a promising approach for loading other large and complex scientific databases.

We present a sample of 218 new quasar pairs with proper transverse separations R_prop < 1 Mpc/h over the redshift range 0.5 < z < 3.0, discovered from an extensive follow up campaign to find companions around the Sloan Digital Sky Survey and 2dF Quasar Redshift Survey quasars. This sample includes 26 new binary quasars with separations R_prop < 50 kpc/h (theta < 10 arcseconds), more than doubling the number of such systems known. We define a statistical sample of binaries selected with homogeneous criteria and compute its selection function, taking into account sources of incompleteness. The first measurement of the quasar correlation function on scales 10 kpc/h < R_prop < 400 kpc/h is presented. For R_prop < 40 kpc/h, we detect an order of magnitude excess clustering over the expectation from the large scale R_prop > 3 Mpc/h quasar correlation function, extrapolated down as a power law to the separations probed by our binaries. The excess grows to ~ 30 at R_prop ~ 10 kpc/h, and provides compelling evidence that the quasar autocorrelation function gets progressively steeper on sub-Mpc scales. This small scale excess can likely be attributed to dissipative interaction events which trigger quasar activity in rich environments. Recent small scale measurements of galaxy clustering and quasar-galaxy clustering are reviewed and discussed in relation to our measurement of small scale quasar clustering.

We present an 8 sigma detection of cosmic magnification measured by the variation of quasar density due to gravitational lensing by foreground large scale structure. To make this measurement we used 3800 square degrees of photometric observations from the Sloan Digital Sky Survey (SDSS) containing \~200,000 quasars and 13 million galaxies. Our measurement of the galaxy-quasar cross-correlation function exhibits the amplitude, angular dependence and change in sign as a function of the slope of the observed quasar number counts that is expected from magnification bias due to weak gravitational lensing. We show that observational uncertainties (stellar contamination, Galactic dust extinction, seeing variations and errors in the photometric redshifts) are well controlled and do not significantly affect the lensing signal. By weighting the quasars with the number count slope, we combine the cross-correlation of quasars for our full magnitude range and detect the lensing signal at >4 sigma in all five SDSS filters. Our measurements of cosmic magnification probe scales ranging from 60 kpc/h to 10 Mpc/h and are in good agreement with theoretical predictions based on the WMAP concordance cosmology. As with galaxy-galaxy lensing, future measurements of cosmic magnification will provide useful constraints on the galaxy-mass power spectrum.

Sloan Digital Sky Survey (SDSS) repeat spectroscopic observations have resulted in multiple-epoch spectroscopy for roughly 2500 quasars observed more than 50 days apart. From this sample, we identify 315 quasars that have varied significantly between observations. We create an ensemble difference spectrum (bright phase minus faint phase) covering rest-frame wavelengths from 1000 to 6000 Angstroms. This average difference spectrum is bluer than the average single-epoch quasar spectrum; a power-law fit to the difference spectrum yields a spectral index alpha_lambda = -2.00, compared to an index of alpha_lambda = -1.35 for the single-epoch spectrum. The strongest emission lines vary only 30% as much as the continuum. Due to the lack of variability of the lines, measured photometric color is not always bluer in brighter phases, but depends on redshift and the filters used. Lastly, the difference spectrum is bluer than the ensemble quasar spectrum only for lambda_rest < 2500 Angstroms, indicating that the variability cannot result from a simple scaling of the average quasar spectrum.

We present the third edition of the Sloan Digital Sky Survey (SDSS) Quasar Catalog. The catalog consists of the 46,420 objects in the SDSS Third Data Release that have luminosities larger than M_i = -22 (in a cosmology with H_0 = 70 km/s/Mpc, Omega_M = 0.3, and Omega_Lambda = 0.7), have at least one emission line with FWHM larger than 1000 km/s or are unambiguously broad absorption line quasars, are fainter than i = 15.0, and have highly reliable redshifts. The area covered by the catalog is 4188 sq. deg. The quasar redshifts range from 0.08 to 5.41, with a median value of 1.47; the high-redshift sample includes 520 quasars at redshifts greater than four, of which 17 are at redshifts greater than five. For each object the catalog presents positions accurate to better than 0.2 arcsec. rms per coordinate, five-band (ugriz) CCD-based photometry with typical accuracy of 0.03 mag, and information on the morphology and selection method. The catalog also contains radio, near-infrared, and X-ray emission properties of the quasars, when available, from other large-area surveys. The calibrated digital spectra cover the wavelength region 3800--9200A at a spectral resolution about 2000; the spectra can be retrieved from the public database using the information provided in the catalog. A total of 44,221 objects in the catalog were discovered by the SDSS; 28,400 of the SDSS discoveries are reported here for the first time.

The Deep Extragalactic Exploratory Probe (DEEP) is a multi-phase research program dedicated to the study of the formation and evolution of galaxies and of large scale structure in the distant Universe. This paper describes the first five-year phase, denoted DEEP1. A series of ten DEEP1 papers will discuss a range of scientific topics (e.g., the study of photometric and spectral properties of a general distant galaxy survey, the evolution observed in galaxy populations of varied morphologies). The observational basis for these studies is the Groth Survey Strip field, a 127 square arcminute region which has been observed with the Hubble Space Telescope in both broad I-band and V-band optical filters and with the Low Resolution Imaging Spectrograph on the Keck Telescopes. Catalogs of photometric and structural parameters have been constructed for 11,547 galaxies and stars at magnitudes brighter than 29, and spectroscopy has been conducted for a magnitude-color weighted subsample of 818 objects. We evaluate three independent techniques for constructing an imaging catalog for the field from the HST data, and discuss the depth and sampling of the resultant catalogs. The selection of the spectroscopic subsample is discussed, and we describe the multifaceted approach taken to prioritizing objects of interest for a variety of scientific subprograms. A series of Monte Carlo simulations then demonstrates that the spectroscopic subsample can be adequately modeled as a simple function of magnitude and color cuts in the imaging catalog.

Spectra of nearly 20000 point-like objects to a Galactic reddening corrected magnitude of i=19.1 have been obtained to test the completeness of the SDSS quasar survey. The spatially-unresolved objects were selected from all regions of color space, sparsely sampled from within a 278 sq. deg. area of sky covered by this study. Only ten quasars were identified that were not targeted as candidates by the SDSS quasar survey (including both color and radio source selection). The inferred density of unresolved quasars on the sky that are missed by the SDSS algorithm is 0.44 per sq. deg, compared to 8.28 per sq. deg. for the selected quasar density, giving a completeness of 94.9(+2.6,-3.8) to the limiting magnitude. Omitting radio selection reduces the color-only selection completeness by about 1%. Of the ten newly identified quasars, three have detected broad absorption line systems, six are significantly redder than other quasars at the same redshift, and four have redshifts between 2.7 and 3.0 (the redshift range where the SDSS colors of quasars intersect the stellar locus). The fraction of quasars missed due to image defects and blends is approximately 4%, but this number varies by a few percent with magnitude. Quasars with extended images comprise about 6% of the SDSS sample, and the completeness of the selection algorithm for extended quasars is approximately 81%, based on the SDSS galaxy survey. The combined end-to-end completeness for the SDSS quasar survey is approximately 89%. The total corrected density of quasars on the sky to i=19.1 is estimated to be 10.2 per sq. deg.

We have compiled a large sample of low-redshift active galactic nuclei (AGN) identified via their emission line characteristics from the spectroscopic data of the Sloan Digital Sky Survey. Since emission lines are often contaminated by stellar absorption lines, we developed an objective and efficient method of subtracting the stellar continuum from every galaxy spectrum before making emission line measurements. The distribution of the measured H$\alpha$ Full Width at Half Maxima values of emission line galaxies is strongly bimodal, with two populations separated at about 1,200km s$^{-1}$. This feature provides a natural separation between narrow-line and broad-line AGN. The narrow-line AGN are identified using standard emission line ratio diagnostic diagrams. 1,317 broad-line and 3,074 narrow-line AGN are identified from about 100,000 galaxy spectra selected over 1151 square degrees. This sample is used in a companion paper to determine the emission-line luminosity function of AGN.

The emission line luminosity function of active galactic nuclei (AGN) is measured from about 3000 AGN included in the main galaxy sample of the Sloan Digital Sky Survey within a redshift range of $0<z<0.15$. The $\Ha$ and [OIII]$\lambda 5007$ luminosity functions for Seyferts cover luminosity range of $10^{5-9}$$L_\odot$ in H$\alpha$ and the shapes are well fit by broken power laws, without a turnover at fainter nuclear luminosities. Assuming a universal conversion from emission line strength to continuum luminosity, the inferred B band magnitude luminosity function is comparable both to the AGN luminosity function of previous studies and to the low redshift quasar luminosity function derived from the 2dF redshift survey. The inferred AGN number density is approximately 1/5 of all galaxies and about $6\times 10^{-3}$ of the total light of galaxies in the $r$-band comes from the nuclear activity. The numbers of Seyfert 1s and Seyfert 2s are comparable at low luminosity, while at high luminosity, Seyfert 1s outnumber Seyfert 2s by a factor of 2-4. In making the luminosity function measurements, we assumed that the nuclear luminosity is independent of the host galaxy luminosity, an assumption we test {\it a posteriori}, and show to be consistent with the data. Given the relationship between black hole mass and host galaxy bulge luminosity, the lack of correlation between nuclear and host luminosity suggests that the main variable that determines the AGN luminosity is the Eddington ratio, not the black hole mass. This appears to be different from luminous quasars, which are most likely to be shining near the Eddington limit.

We report the discovery of two doubly-imaged quasars, SDSS J100128.61+502756.9 and SDSS J120629.65+433217.6, at redshifts of 1.838 and 1.789 and with image separations of 2.86'' and 2.90'', respectively. The objects were selected as lens candidates from the Sloan Digital Sky Survey (SDSS). Based on the identical nature of the spectra of the two quasars in each pair and the identification of the lens galaxies, we conclude that the objects are gravitational lenses. The lenses are complicated; in both systems there are several galaxies in the fields very close to the quasars, in addition to the lens galaxies themselves. The lens modeling implies that these nearby galaxies contribute significantly to the lens potentials. On larger scales, we have detected an enhancement in the galaxy density near SDSS J100128.61+502756.9. The number of lenses with image separation of ~3'' in the SDSS already exceeds the prediction of simple theoretical models based on the standard Lambda-dominated cosmology and observed velocity function of galaxies.

One of the new frontiers of astronomical research is the exploration of time variability on the sky at different wavelengths and flux levels. We have carried out a pilot project using DPOSS data to study strong variables and transients, and are now extending it to the new Palomar-QUEST synoptic sky survey. We report on our early findings and outline the methodology to be implemented in preparation for a real-time transient detection pipeline. In addition to large numbers of known types of highly variable sources (e.g., SNe, CVs, OVV QSOs, etc.), we expect to find numerous transients whose nature may be established by a rapid follow-up. Whereas we will make all detected variables publicly available through the web, we anticipate that email alerts would be issued in the real time for a subset of events deemed to be the most interesting. This real-time process entails many challenges, in an effort to maintain a high completeness while keeping the contamination low. We will utilize distributed Grid services developed by the GRIST project, and implement a variety of advanced statistical and machine learning techniques.

We present a catalog of 100,563 unresolved, UV-excess (UVX) quasar candidates to g=21 from 2099 deg^2 of the Sloan Digital Sky Survey (SDSS) Data Release One (DR1) imaging data. Existing spectra of 22,737 sources reveals that 22,191 (97.6%) are quasars; accounting for the magnitude dependence of this efficiency, we estimate that 95,502 (95.0%) of the objects in the catalog are quasars. Such a high efficiency is unprecedented in broad-band surveys of quasars. This ``proof-of-concept'' sample is designed to be maximally efficient, but still has 94.7% completeness to unresolved, g<~19.5, UVX quasars from the DR1 quasar catalog. This efficient and complete selection is the result of our application of a probability density type analysis to training sets that describe the 4-D color distribution of stars and spectroscopically confirmed quasars in the SDSS. Specifically, we use a non-parametric Bayesian classification, based on kernel density estimation, to parameterize the color distribution of astronomical sources -- allowing for fast and robust classification. We further supplement the catalog by providing photometric redshifts and matches to FIRST/VLA, ROSAT, and USNO-B sources. Future work needed to extend the this selection algorithm to larger redshifts, fainter magnitudes, and resolved sources is discussed. Finally, we examine some science applications of the catalog, particularly a tentative quasar number counts distribution covering the largest range in magnitude (14.2<g<21.0) ever made within the framework of a single quasar survey.

Exploration of the time variability on the sky over a broad range of flux levels and wavelengths is rapidly becoming a new frontier of astronomical research. We describe here briefly the Palomar-QUEST survey being carried out from the Samuel Oschin 48-inch Schmidt telescope at Palomar. The following features make the survey an attractive candidate for studying time variability: anticipated survey area of 12,000 - 15,000 sq. degrees in the drift scan mode, point source depth of 21st mag. in I under good conditions, near simultaneous observations in four filters, and at least four passes per year at each location covered. The survey will yield a large number of transients and highly variable sources in the near future and in that sense is a prototype of LSST and Pan-STARRS. We briefly outline our strategy for searching such objects and the proposed pipeline for detecting transients in real-time.

We present an optically selected galaxy cluster catalog from ~ 2,700 square degrees of the Digitized Second Palomar Observatory Sky Survey (DPOSS), spanning the redshift range 0.1 < z < 0.5, providing an intermediate redshift supplement to the previous DPOSS cluster survey. This new catalog contains 9,956 cluster candidates and is the largest resource of rich clusters in this redshift range to date. The candidates are detected using the best DPOSS plates based on seeing and limiting magnitude. The search is further restricted to high galactic latitude (|b| > 50), where stellar contamination is modest and nearly uniform. We also present a performance comparison of two different detection methods applied to this data, the Adaptive Kernel and Voronoi Tessellation techniques. In the regime where both catalogs are expected to be complete, we find excellent agreement, as well as with the most recent surveys in the literature. Extensive simulations are performed and applied to the two different methods, indicating a contamination rate of ~ 5%. These simulations are also used to optimize the algorithms and evaluate the selection function for the final cluster catalog. Redshift and richness estimates are also provided, making possible the selection of subsamples for future studies.

We measure the power spectrum, P_F(k,z), of the transmitted flux in the Ly-alpha forest using 3035 high redshift quasar spectra from the Sloan Digital Sky Survey. This sample is almost two orders of magnitude larger than any previously available data set, yielding statistical errors of ~0.6% and ~0.005 on, respectively, the overall amplitude and logarithmic slope of P_F(k,z). This unprecedented statistical power requires a correspondingly careful analysis of the data and of possible systematic contaminations in it. For this purpose we reanalyze the raw spectra to make use of information not preserved by the standard pipeline. We investigate the details of the noise in the data, resolution of the spectrograph, sky subtraction, quasar continuum, and metal absorption. We find that background sources such as metals contribute significantly to the total power and have to be subtracted properly. We also find clear evidence for SiIII correlations with the Ly-alpha forest and suggest a simple model to account for this contribution to the power. While it is likely that our newly developed analysis technique does not eliminate all systematic errors in the P_F(k,z) measurement below the level of the statistical errors, our tests indicate that any residual systematics in the analysis are unlikely to affect the inference of cosmological parameters from P_F(k,z). These results should provide an essential ingredient for all future attempts to constrain modeling of structure formation, cosmological parameters, and theories for the origin of primordial fluctuations.

We present a study of variable faint optical sources discovered by comparing the Sloan Digital Sky Survey (SDSS) and the Palomar Observatory Sky Survey (POSS) catalogs. We use SDSS measurements to photometrically recalibrate several publicly available POSS catalogs; a piecewise recalibration in 100 arcmin2 patches generally results in an improvement of photometric accuracy (rms) by nearly a factor of two, compared to the original data. The POSS I magnitudes can be improved to ~0.15 mag accuracy, and POSS II magnitudes to \~0.10 mag accuracy. We use the recalibrated catalogs for the ~2,000 deg2 of sky in the SDSS Data Release 1 to construct a catalog of ~60,000 sources variable on time scales 10-50 years. A series of statistical tests based on the morphology of SDSS color-color diagrams, as well as visual comparison of images and comparison with repeated SDSS observations, demonstrate the robustness of the selection methods. We quantify the distribution of variable sources in the SDSS color-color diagrams, and the variability characteristics of quasars. We detect a turn-over in quasar structure function which suggests that the characteristic time scale for quasar variability is of the order one year. The long-term (>1 year) quasar variability decreases with luminosity and rest-frame wavelength similarly to the short-term (<1 year) behavior. We also demonstrate that candidate RR Lyrae stars trace the same halo structures, such as the Sgr dwarf tidal stream, that were discovered using repeated SDSS observations. We utilize the POSS-SDSS selected candidates to constrain the halo structure in the parts of sky for which repeated SDSS observations do not exist. (abridged)

The advent of wide-area multicolour synoptic sky surveys is leading to data sets unprecedented in size, complexity and data throughput. VO technology offers a way to exploit these to the full but requires changes in design philosophy. The Palomar-QUEST survey is a major new survey being undertaken by Caltech, Yale, JPL and Indiana University to repeatedly observe 1/3 of the sky (~15000 sq. deg. between -27 < Dec <27 in seven passbands. Utilising the 48-inch Oschin Schmidt Telescope at the Palomar Observatory with the 112-CCD QUEST camera covering the full 4 x 4 sq. deg. field of view, it will generate \~1TB of data per month. In this paper, we review the design of QUEST as a VO resource, a federated data set and an exemplar of VO standards.

Using a sample of over 25000 spectroscopically confirmed quasars from the Sloan Digital Sky Survey, we show how quasar variability in the rest frame optical/UV regime depends upon rest frame time lag, luminosity, rest wavelength, redshift, the presence of radio and X-ray emission, and the presence of broad absorption line systems. The time dependence of variability (the structure function) is well-fit by a single power law on timescales from days to years. There is an anti-correlation of variability amplitude with rest wavelength, and quasars are systematically bluer when brighter at all redshifts. There is a strong anti-correlation of variability with quasar luminosity. There is also a significant positive correlation of variability amplitude with redshift, indicating evolution of the quasar population or the variability mechanism. We parameterize all of these relationships. Quasars with RASS X-ray detections are significantly more variable (at optical/UV wavelengths) than those without, and radio loud quasars are marginally more variable than their radio weak counterparts. We find no significant difference in the variability of quasars with and without broad absorption line troughs. Models involving multiple discrete events or gravitational microlensing are unlikely by themselves to account for the data. So-called accretion disk instability models are promising, but more quantitative predictions are needed.

Over the last few years, the Sloan Digital Sky Survey (SDSS) has discovered several hundred quasars with redshift between 4.0 and 6.4. Including the effects of magnification bias, one expects a priori that an appreciable fraction of these objects are gravitationally lensed. We have used the Advanced Camera for Surveys on the Hubble Space Telescope to carry out a snapshot imaging survey of high-redshift SDSS quasars to search for gravitationally split lenses. This paper, the first in a series reporting the results of the survey, describes snapshot observations of four quasars at z = 5.74, 5.82, 5.99 and 6.30, respectively. We find that none of these objects has a lensed companion within 5 magnitudes with a separation larger than 0.3 arcseconds; within 2.5 magnitudes, we can rule out companions within 0.1 arcseconds. Based on the non-detection of strong lensing in these four systems, we constrain the z~6 luminosity function to a slope of beta>-4.63 (3 sigma), assuming a break in the quasar luminosity function at M_{1450}^*=-24.0. We discuss the implications of this constraint on the ionizing background due to quasars in the early universe. Given that these quasars are not highly magnified, estimates of the masses of their central engines by the Eddington argument must be taken seriously, possibly challenging models of black hole formation.

We report a discovery of a quasar at z = 4.96 +- 0.03 within a few Mpc of the quasar SDSS 0338+0021 at z = 5.02 +- 0.02. The newly found quasar has the SDSS i and z magnitudes of ~ 21.2, and an estimated absolute magnitude M_B ~ -25.2. The projected separation on the sky is 196 arcsec, and the redshift difference Delta z = 0.063 +- 0.008. The probability of finding this quasar pair by chance in the absence of clustering in this particular volume is ~ 10^-4 to 10^-3. We conclude that the two objects probably mark a large-scale structure, possibly a protocluster, at z ~ 5. This is the most distant such structure currently known. Our search in the field of 13 other QSOs at z >~ 4.8 so far has not resulted in any detections of comparable luminous QSO pairs, and it is thus not yet clear how representative is this structure at z ~ 5. However, along with the other evidence for clustering of quasars and young galaxies at somewhat lower redshifts, the observations are at least qualitatively consistent with a strong biasing of the first luminous and massive objects, in agreement with general predictions of theoretical models. More extensive searches for clustered quasars and luminous galaxies at these redshifts will provide valuable empirical constraints for our understanding of early galaxy and structure formation.

Supervised artificial neural networks are used to predict useful properties of galaxies in the Sloan Digital Sky Survey, in this instance morphological classifications, spectral types and redshifts. By giving the trained networks unseen data, it is found that correlations between predicted and actual properties are around 0.9 with rms errors of order ten per cent. Thus, given a representative training set, these properties may be reliably estimated for galaxies in the survey for which there are no spectra and without human intervention.

With the recent release of large (i.e., > hundred million objects), well-calibrated photometric surveys, such as DPOSS, 2MASS, and SDSS, spectroscopic identification of important targets is no longer a simple issue. In order to enhance the returns from a spectroscopic survey, candidate sources are often preferentially selected to be of interest, such as brown dwarfs or high redshift quasars. This approach, while useful for targeted projects, risks missing new or unusual species. We have, as a result, taken the alternative path of spectroscopically identifying interesting sources with the sole criterion being that they are in low density areas of the g - r and r - i color-space defined by the DPOSS survey. In this paper, we present three peculiar broad absorption line quasars that were discovered during this spectroscopic survey, demonstrating the efficacy of this approach. PSS J0052+2405 is an Iron LoBAL quasar at a redshift z = 2.4512 with very broad absorption from many species. PSS J0141+3334 is a reddened LoBAL quasar at z = 3.005 with no obvious emission lines. PSS J1537+1227 is a Iron LoBAL at a redshift of z = 1.212 with strong narrow Mgii and Feii emission. Follow-up high resolution spectroscopy of these three quasars promises to improve our understanding of BAL quasars. The sensitivity of particular parameter spaces, in this case a two-color space, to the redshift of these three sources is dramatic, raising questions about traditional techniques of defining quasar populations for statistical analysis.

We present a new, objectively defined catalog of candidate galaxy clusters based on the galaxy catalogs from the Digitized Second Palomar Observatory Sky Survey (DPOSS). This cluster catalog, derived from the best calibrated plates in the high latitude (|b|>30) Northern Galactic Cap region, covers 5,800 square degrees, and contains 8,155 candidate clusters. A simple adaptive kernel density mapping technique, combined with the SExtractor object detection algorithm, is used to detect galaxy overdensities, which we identify as clusters. Simulations of the background galaxy distribution and clusters of varying richnesses and redshifts allow us to optimize detection parameters, and measure the completeness and contamination rates for our catalog. Cluster richnesses and photometric redshifts are measured, using integrated colors and magnitudes for each cluster. An extensive spectroscopic survey is used to confirm the photometric results. This catalog, with well-characterized sample properties, provides a sound basis for future studies of cluster physics and large scale structure.

A Topic Map is a structured network of hyperlinks that points into an information pool. Topic Maps have an existence independent of the information pool and hence different Topic Maps can form different layers above the same information pool and provide us with different views of it. We explore the use of Topic Maps with the Unified Column Descriptor (UCD) scheme developed in the frame of the ESO-CDS data mining project. UCD, with its multi-tier hierarchical structure, categorizes parameters reported in tables and catalogs. By using Topic Maps we show how columns from different catalogs with similar but not identical descriptions could be combined. A direct application for the Virtual Observatory community is that of merging catalogs in order to generate customized views of data.

We present the photometric calibration technique for the Digitized Second Palomar Observatory Sky Survey (DPOSS), used to create seamless catalogs of calibrated objects over large sky areas. After applying a correction for telescope vignetting, the extensive plate overlap regions are used to transform sets of plates onto a common instrumental photometric system. Photometric transformations to the Gunn gri system for each plate, for stars and galaxies, are derived using these contiguous stitched areas and an extensive CCD imaging library obtained for this purpose. We discuss the resulting photometric accuracy, survey depth, and possible systematic errors.

There has been an unprecedented and continuing growth in the volume, quality, and complexity of astronomical data sets over the past few years, mainly through large digital sky surveys. Virtual Observatory (VO) concept represents a scientific and technological framework needed to cope with this data flood. We review some of the applied statistics and computing challenges posed by the analysis of large and complex data sets expected in the VO-based research. The challenges are driven both by the size and the complexity of the data sets (billions of data vectors in parameter spaces of tens or hundreds of dimensions), by the heterogeneity of the data and measurement errors, the selection effects and censored data, and by the intrinsic clustering properties (functional form, topology) of the data distribution in the parameter space of observed attributes. Examples of scientific questions one may wish to address include: objective determination of the numbers of object classes present in the data, and the membership probabilities for each source; searches for unusual, rare, or even new types of objects and phenomena; discovery of physically interesting multivariate correlations which may be present in some of the clusters; etc.

We report on exploratory Chandra observations of the three highest redshift quasars known (z = 5.82, 5.99, and 6.28), all found in the Sloan Digital Sky Survey. These data, combined with a previous XMM-Newton observation of a z = 5.74 quasar, form a complete set of color-selected, z > 5.7 quasars. X-ray emission is detected from all of the quasars at levels that indicate that the X-ray to optical flux ratios of z ~ 6 optically selected quasars are similar to those of lower redshift quasars. The observations demonstrate that it will be feasible to obtain quality X-ray spectra of z ~ 6 quasars with current and future X-ray missions.

We present a comparison of three cluster finding algorithms from imaging data using Monte Carlo simulations of clusters embedded in a 25 deg^2 region of Sloan Digital Sky Survey (SDSS) imaging data: the Matched Filter (MF; Postman et al. 1996), the Adaptive Matched Filter (AMF; Kepner et al. 1999) and a color-magnitude filtered Voronoi Tessellation Technique (VTT). Among the two matched filters, we find that the MF is more efficient in detecting faint clusters, whereas the AMF evaluates the redshifts and richnesses more accurately, therefore suggesting a hybrid method (HMF) that combines the two. The HMF outperforms the VTT when using a background that is uniform, but it is more sensitive to the presence of a non-uniform galaxy background than is the VTT; this is due to the assumption of a uniform background in the HMF model. We thus find that for the detection thresholds we determine to be appropriate for the SDSS data, the performance of both algorithms are similar; we present the selection function for each method evaluated with these thresholds as a function of redshift and richness. For simulated clusters generated with a Schechter luminosity function (M_r^* = -21.5 and alpha = -1.1) both algorithms are complete for Abell richness >= 1 clusters up to z ~ 0.4 for a sample magnitude limited to r = 21. While the cluster parameter evaluation shows a mild correlation with the local background density, the detection efficiency is not significantly affected by the background fluctuations, unlike previous shallower surveys.

One major component of the VO will be catalogs measuring gigabytes and terrabytes if not more. Some mechanism like XML will be used for structuring the information. However, such mechanisms are not good for information retrieval on their own. For retrieval we use queries. Topic Maps that have started becoming popular recently are excellent for segregating information that results from a query. A Topic Map is a structured network of hyperlinks above an information pool. Different Topic Maps can form different layers above the same information pool and provide us with different views of it. This facilitates in being able to ask exact questions, aiding us in looking for gold needles in the proverbial haystack. Here we discuss the specifics of what Topic Maps are and how they can be implemented within the VO framework. URL: http://www.astro.caltech.edu/~aam/science/topicmaps/

As a scientific discipline, Astronomy is rather unique. We only have one laboratory, the Universe, and we cannot, of course, change the initial conditions and study the resulting effects. On top of this, acquiring Astronomical data has historically been a very labor-intensive effort. As a result, data has traditionally been preserved for posterity. With recent technological advances, however, the rate at which we acquire new data has grown exponentially, which has generated a Data Tsunami, whose wave train threatens to overwhelm the field. In this conference proceedings, we present and define the concept of virtual observatories, which we feel is the only logical answer to this dilemma.

A data Tsunami is overwhelming Astronomy. This wave is affecting all aspects of our field, revolutionizing not just the type of scientific questions being asked, but the very nature of how the answers are uncovered. In this invited proceeding, we will address a particular scientific application - Panchromatic Mining for Quasars - of the forthcoming virtual observatories, which have arisen in an effort to control the effects of the data Tsunami. This project, in addition to serving as an important scientific driver for virtual observatory technologies, is designed to a) characterize the multi-wavelength nature of known active galaxies and quasars, especially in relation to their local environment, in order to b) quantify the clustering of these known systems in the multidimensional parameter space formed by their observables, so that new, and potentially unknown types of systems can be optimally targeted.

Like every other field of intellectual endeavor, astronomy is being revolutionised by the advances in information technology. There is an ongoing exponential growth in the volume, quality, and complexity of astronomical data sets, mainly through large digital sky surveys and archives. The Virtual Observatory (VO) concept represents a scientific and technological framework needed to cope with this data flood. Systematic exploration of the observable parameter spaces, covered by large digital sky surveys spanning a range of wavelengths, will be one of the primary modes of research with a VO. This is where the truly new discoveries will be made, and new insights be gained about the already known astronomical objects and phenomena. We review some of the methodological challenges posed by the analysis of large and complex data sets expected in the VO-based research. The challenges are driven both by the size and the complexity of the data sets (billions of data vectors in parameter spaces of tens or hundreds of dimensions), by the heterogeneity of the data and measurement errors, including differences in basic survey parameters for the federated data sets (e.g., in the positional accuracy and resolution, wavelength coverage, time baseline, etc.), various selection effects, as well as the intrinsic clustering properties (functional form, topology) of the data distributions in the parameter spaces of observed attributes. Answering these challenges will require substantial collaborative efforts and partnerships between astronomers, computer scientists, and statisticians.

The Sloan Digital Sky Survey has discovered a population of broad absorption line quasars with various extreme properties. Many show absorption from metastable states of FeII with varying excitations; several objects are almost completely absorbed bluewards of MgII; at least one shows stronger absorption from FeIII than FeII, indicating temperatures T>35000 K in the absorbing region; and one object even seems to have broad H-beta absorption. Many of these extreme BALs are also heavily reddened, though `normal' BALs (particularly LoBALs) from SDSS also show evidence for internal reddening.

Astronomy has a long history of acquiring, systematizing, and interpreting large quantities of data. Starting from the earliest sky atlases through the first major photographic sky surveys of the 20th century, this tradition is continuing today, and at an ever increasing rate. Like many other fields, astronomy has become a very data-rich science, driven by the advances in telescope, detector, and computer technology. Numerous large digital sky surveys and archives already exist, with information content measured in multiple Terabytes, and even larger, multi-Petabyte data sets are on the horizon. Systematic observations of the sky, over a range of wavelengths, are becoming the primary source of astronomical data. Numerical simulations are also producing comparable volumes of information. Data mining promises to both make the scientific utilization of these data sets more effective and more complete, and to open completely new avenues of astronomical research. Technological problems range from the issues of database design and federation, to data mining and advanced visualization, leading to a new toolkit for astronomical research. This is similar to challenges encountered in other data-intensive fields today. These advances are now being organized through a concept of the Virtual Observatories, federations of data archives and services representing a new information infrastructure for astronomy of the 21st century. In this article, we provide an overview of some of the major datasets in astronomy, discuss different techniques used for archiving data, and conclude with a discussion of the future of massive datasets in astronomy.

We present a lensing study of 42 galaxy clusters imaged in Sloan Digital Sky Survey (SDSS) commissioning data. Cluster candidates are selected optically from SDSS imaging data and confirmed for this study by matching to X-ray sources found independently in the ROSAT all sky survey (RASS). Five color SDSS photometry is used to make accurate photometric redshift estimates that are used to rescale and combine the lensing measurements. The mean shear from these clusters is detected to 2 h-1 Mpc at the 7-sigma level, corresponding to a mass within that radius of 4.2 +/- 0.6 x 10^14 h-1 M_sun. The shear profile is well fit by a power law with index -0.9 +/- 0.3, consistent with that of an isothermal density profile. This paper demonstrates our ability to measure ensemble cluster masses from SDSS imaging data.

We review some of the scientific opportunities and technical challenges posed by the exploration of the large digital sky surveys, in the context of a Virtual Observatory (VO). The VO paradigm will profoundly change the way observational astronomy is done. Clustering analysis techniques can be used to discover samples of rare, unusual, or even previously unknown types of astronomical objects and phenomena. Exploration of the previously poorly probed portions of the observable parameter space are especially promising. We illustrate some of the possible types of studies with examples drawn from DPOSS; much more complex and interesting applications are forthcoming. Development of the new tools needed for an efficient exploration of these vast data sets requires a synergy between astronomy and information sciences, with great potential returns for both fields.

Systematic exploration of the observable parameter space, covered by large digital sky surveys spanning a range of wavelengths, will be one of the primary modes of research with a Virtual Observatory (VO). This will include searches for rare, unusual, or even previously unknown types of astronomical objects and phenomena, e.g. as outliers in some parameter space of measured properties, both in the catalog and image domains. Examples from current surveys include high-redshift quasars, type-2 quasars, brown dwarfs, and a small number of objects with puzzling spectra. Opening of the time domain will be especially interesting in this regard. Data-mining tools such as unsupervised clustering techniques will be essential in this task, and should become an important part of the VO toolkit.

A virtual observatory will not only enhance many current scientific investigations, but it will also enable entirely new scientific explorations due to both the federation of vast amounts of multiwavelength data and the new archival services which will, as a necessity, be developed. The detailing of specific science use cases is important in order to properly facilitate the development of the necessary infrastructure of a virtual observatory. The understanding of high velocity clouds is presented as an example science use case, demonstrating the future synergy between the data (either catalog or images), and the desired analysis in the new paradigm of a virtual observatory.

Astronomy is entering a new era as multiple, large area, digital sky surveys are in production. The resulting datasets are truly remarkable in their own right; however, a revolutionary step arises in the aggregation of complimentary multi-wavelength surveys (i.e., the cross-identification of a billion sources). The federation of these large datasets is already underway, and is producing a major paradigm shift as Astronomy has suddenly become an immensely data-rich field. This new paradigm will enable quantitatively and qualitatively new science, from statistical studies of our Galaxy and the large-scale structure in the universe, to discoveries of rare, unusual, or even completely new types of astronomical objects and phenomena. Federating and then exploring these large datasets, however, is an extremely challenging task. The Digital Sky project was initiated with this task in mind and is working to develop the techniques and technologies necessary to solve the problems inherent in federating these large databases, as well as the mining of the resultant aggregate data.

Our current understanding of the low surface brightness universe is quite incomplete, not only in the optical, but also in other wavelength regimes. As a demonstration of the type of science which is facilitated by a virtual observatory, we have undertaken a project utilizing both images and catalogs to explore the multi-wavelength, low surface brightness universe. Here, we present some initial results of this project. Our techniques are complimentary to normal data reduction pipeline techniques in that we focus on the diffuse emission that is ignored or removed by more traditional algorithms. This requires a spatial filtering which must account for objects of interest, in addition to observational artifacts (e.g., bright stellar halos). With this work we are exploring the intersection of the catalog and image domains in order to maximize the scientific information we can extract from the federation of large survey data.

Clusters of galaxies are important laboratories for understanding both galaxy evolution and constraining cosmological quantities. Any analysis of clusters, however, is best done when one can reliably determine which galaxies are members of the cluster. While this would ideally be done spectroscopically, the difficulty in acquiring a complete sample of spectroscopic redshifts becomes rather daunting, especially at high redshift where the background contamination becomes increasingly larger. Traditionally, an alternative approach of applying a statistical background correction has been utilized, which, while useful in a global sense, does not provide information for specific galaxies. In this paper, we develop a more robust technique which uses photometrically estimated redshifts to determine cluster membership. This technique can either be used as an improvement over the commonly used statistical correction method or it can be used to determine cluster candidates on an individual galaxy basis. By tuning the parameters of our algorithm, we can selectively maximize our completeness or, alternatively, minimize our contamination. Furthermore, our technique provides a statistical quantification of both our resulting completeness and contamination from foreground and background galaxies.

We report the discovery of a pair of z = 4.25 quasars with a separation of 33 arcseconds. The brighter of the two objects was identified as a high-redshift quasar candidate from Sloan Digital Sky Survey multicolor imaging data, and the redshift was measured from a spectrum obtained with the Hobby-Eberly Telescope. The slit orientation of this observation {\it by chance} included another quasar, approximately one magnitude fainter and having the same redshift as the target. This is the third serendipitous discovery of a z > 4 quasar. The differences in the relative strengths and profiles of the emission lines suggest that this is a quasar pair and not a gravitational lens. The two objects are likely to be physically associated; the projected physical separation is approximately 210 $h_{50}^{-1}$ kpc and the redshifts are identical to $\approx$ 0.01, implying a radial physical separation of 950 $h_{50}^{-1}$ kpc or less. The existence of this pair is strong circumstantial evidence that $z \sim 4$ quasars are clustered.

In this paper we explore the application of cryogenic imaging spectrophotometers. Prototypes of this new class of detector, such as superconducting tunnel junctions (STJs) and transition edge sensors (TESs), currently deliver low resolution imaging spectrophotometry with high quantum efficiency (70-100%) and no read noise over a wide bandpass in the visible to near-infrared. In order to demonstrate their utility and the differences in observing strategy needed to maximize their scientific return, we present simulated observations of a deep extragalactic field. Using a simple analytic technique, we can estimate both the galaxy redshift and spectral type more accurately than is possible with current broadband techniques. From our simulated observations and a subsequent discussion of the expected migration path for this new technology, we illustrate the power and promise of these devices.

Large digital sky surveys, over a broad range of wavelengths, both from the ground and from space observatories, are becoming a major source of astronomical data. Some examples include the Sloan Digital Sky Survey (SDSS) and the Digital Palomar Observatory Sky Survey (DPOSS) in the visible, the Two-Micron All-Sky Survey (2MASS) in the near-infrared, the NRAO VLA Sky Survey (NVSS) and the Faint Images of the Radio Sky at Twenty centimeters (FIRST) in the radio. Many others surveys are planned or expected, in addition to the previously named surveys. While most surveys are exclusively imaging, large-scale spectroscopic surveys also exist. In addition, a number of experiments with specific scientific goals, e.g., microlensing surveys for MACHOs, searches for near-Earth asteroids, are generating comparable data volumes. Typical sizes of resulting data sets (as of the late 1990's) are in the range of tens of Terabytes of digital information, with detections of many millions or even billions of sources, and several tens of parameters measured for each detected source. This vast amount of new information presents both a great scientific opportunity and a great technological challenge: how to process, and calibrate the raw data; how to store, combine, and access them using modern computing hardware and networks; and how to visualize, explore and analyses these great data sets quickly and efficiently.

Measuring the evolution in the clustering of galaxies over a large redshift range is a challenging problem. We have developed a new technique which uses photometric redshifts to measure the angular correlation function in redshift shells. This novel approach minimizes the galaxy projection effect inherent in standard angular correlation measurements, and allows for a measurement of the evolution in the galaxy correlation strength with redshift. In this paper, we present new results which utilize more accurate photometric redshifts, which are derived from a multi-band dataset (U,B,R, and I) covering almost two hundred square arcminutes to B_{AB} ~ 26.5, to quantify the evolution in the clustering of galaxies for z < 1. We also extend our technique to incorporate absolute magnitudes, which provides a simultaneous measurement of the evolution of clustering with both redshift and intrinsic luminosity. Specifically, we find a gradual decline in the strength of clustering with redshift out to z ~ 1, as predicted by semi-analytic models of structure formation. Furthermore, we find that r_0(z=0) ~ 4.0 h^{-1} Mpc for the predictions of linear theory in an Omega_0 = 0.1 universe.

This paper presents photometric redshifts for 431 Abell clusters imaged as part of the Palomar Abell Cluster Optical Survey (PACOS), of which 236 are new redshi fts. We have obtained moderately deep, 3--band (Gunn gri) imaging for this sam ple at the Palomar Observatory 60'' telescope, as part of the photometric calibration of DPOSS. Our data acquisition, reduction, and photometric calibration techniques are described, and photometric accuracy and consistency is demonstrated. An empirical redshift estimator is presented, utilizing background-corrected median g-r colors and mean g magnitudes for the ensemble of galaxies in each field. We present photometric redshift estimates for the clusters in our sample with an accuracy of sigma_z=0.038. These redshift estimates provide checks on single-galaxy cluster redshifts, as well as distance information for studies of the Butcher-Oemler effect, luminosity functions, M/L ratios, and many other projects.

We quantitatively cross-associate the 18811 ROSAT Bright Source Catalog (RASS/BSC) X-ray sources with optical sources in the USNO-A2 catalog, calculating the the probability of unique association (Pid) between each candidate within 75 arcsec of the X-ray source position, on the basis of optical magnitude and proximity. We present catalogs of RASS/BSC sources for which the probability of association is >98%, >90%, and >50%, which contain 2705, 5492, and 11301 unique USNO-A2 optical counterparts respectively down to the stated level of significance. We include in this catalog a list of objects in the SIMBAD database within 10 arcsec of the USNO position, as an aid to identification and source classification. The catalog is more useful than previous catalogs which either rely on plausibility arguments for association, or do not aid in selecting a counterpart between multiple off-band sources in the field. We find that a fraction ~65.8% of RASS/BSC sources have an identifiable optical counterpart, down to the magnitude limit of the USNO catalog which could be identified by their spatial proximity and high optical brightness.

We have measured the rest-frame B,V, and I-band light curves of a high-redshift type Ia supernova (SN Ia), SN 1999Q (z=0.46), using HST and ground-based near-infrared detectors. A goal of this study is the measurement of the color excess, E_{B-I}, which is a sensitive indicator of interstellar or intergalactic dust which could affect recent cosmological measurements from high-redshift SNe Ia. Our observations disfavor a 30% opacity of SN Ia visual light by dust as an alternative to an accelerating Universe. This statement applies to both Galactic-type dust (rejected at the 3.4 sigma confidence level) and greyer dust (grain size > 0.1 microns; rejected at the 2.3 to 2.6 sigma confidence level) as proposed by Aguirre (1999). The rest-frame $I$-band light cur ve shows the secondary maximum a month after B maximum typical of nearby SNe Ia of normal luminosi ty, providing no indication of evolution as a function of redshift out to z~0.5. A n expanded set of similar observations could improve the constraints on any contribution of extragalactic dust to the dimming of high-redshift SNe Ia.

We present the results from a multi-band optical imaging program which has definitively confirmed the existence of a supercluster at z = 0.91. Two massive clusters of galaxies, CL1604+4304 at z = 0.897 and CL1604+4321 at z = 0.924, were originally observed in the high-redshift cluster survey of Oke, Postman & Lubin (1998). They are separated by 4300 km/s in radial velocity and 17 arcminutes on the plane of the sky. Their physical and redshift proximity suggested a promising supercluster candidate. Deep BRi imaging of the region between the two clusters indicates a large population of red galaxies. This population forms a tight, red sequence in the color--magnitude diagram at (R-i) = 1.4. The characteristic color is identical to that of the spectroscopically-confirmed early-type galaxies in the two member clusters. The red galaxies are spread throughout the 5 Mpc region between CL1604+4304 and CL1604+4321. Their spatial distribution delineates the entire large scale structure with high concentrations at the cluster centers. In addition, we detect a significant overdensity of red galaxies directly between CL1604+4304 and CL1604+4321 which is the signature of a third, rich cluster associated with this system. The strong sequence of red galaxies and their spatial distribution clearly indicate that we have discovered a supercluster at z = 0.91.

We report on the current status of our search for protoclusters around quasars at z > 4. While the search is still very incomplete, clustered companion galaxies are found in virtually every case examined so far. The implied comoving number densities of protogalaxies are two to four orders of magnitude higher than expected for the general field, but are comparable to the number densities in rich cluster cores. The comoving densities of star formation in these regions are also enhanced by a comparable factor. We interpret these results as an evidence for biased galaxy formation in the highest peaks of the primordial density field.

Measuring the evolution in the clustering of galaxies over a large redshift range is a challenging problem. For a two-dimensional galaxy catalog, however, we can measure the galaxy-galaxy angular correlation function which provides information on the density distribution of galaxies. By utilizing photometric redshifts, we can measure the angular correlation function in redshift shells (Brunner 1997, Connolly et al. 1998) which minimizes the galaxy projection effect, and allows for a measurement of the evolution in the correlation strength with redshift. In this proceedings, we present some preliminary results which extend our previous work using more accurate photometric redshifts, and also incorporate absolute magnitudes, so that we can measure the evolution of clustering with either redshift or intrinsic luminosity.

We present the first results of our application of photometric redshifts to the study of galaxy populations in high-redshift clusters. For this survey, we are examining a sample of galaxy clusters at z > 0.6 which have already been well-studied in the optical and infrared wavelengths (Oke, Postman & Lubin 1998). Our main goal is to use photometric redshifts to delineate accurately between field and cluster galaxies. Once we isolate the cluster galaxies, we can directly study the properties of the galaxy population in each high-redshift cluster. Specifically, we are studying the cluster morphological fractions, the morphology-density relation, and the large scale structure distribution. Although we have encountered some operational problems with our photometric redshift technique, early results suggest that this procedure will become a significant tool for studying high-redshift clusters.

We report on the creation of an unbiased catalog of galaxy clusters from the galaxy catalogs derived from the digitized POSS-II (DPOSS). Utilizing the g-r color information, we show that it is possible to estimate redshifts for galaxy clusters at z<0.4 with an rms accuracy of 0.01.

Astronomy is entering a new era as multiple, large area, digital sky surveys are in production. The resulting datasets are truly remarkable in their own right; however, a revolutionary step arises in the aggregation of complimentary multi-wavelength surveys (i.e. the cross-identification of a billion sources). Federating these different datasets, however, is an extremely challenging task. With this task in mind, we have identified several areas where community standardization can provide enormous benefits in order to develop the techniques and technologies necessary to solve the problems inherent in federating these large databases, as well as the mining of the resultant aggregate data. Several of these areas are domain specific, however, the majority of them are not. We feel that the inclusion of non-astronomical partnerships can provide tremendous insights.

Studies of the distribution and evolution of galaxies are of fundamental importance to modern cosmology; these studies, however, are hampered by the complexity of the competing effects of spectral and density evolution. Constructing a spectroscopic sample that is able to unambiguously disentangle these processes is currently excessively prohibitive due to the observational requirements. This paper extends and applies an alternative approach that relies on statistical estimates for both distance (z) and spectral type to a deep multi-band dataset that was obtained for this exact purpose. These statistical estimates are extracted directly from the photometric data by capitalizing on the inherent relationships between flux, redshift, and spectral type. These relationships are encapsulated in the empirical photometric redshift relation which we extend to z ~ 1.2, with an intrinsic dispersion of dz = 0.06. We also develop realistic estimates for the photometric redshift error for individual objects, and introduce the utilization of the galaxy ensemble as a tool for quantifying both a cosmological parameter and its measured error. We present deep, multi-band, optical number counts as a demonstration of the integrity of our sample. Using the photometric redshift and the corresponding redshift error, we can divide our data into different redshift intervals and spectral types. As an example application, we present the number redshift distribution as a function of spectral type.

We describe DPOSS, a new digital survey of the northern sky, based on the POSS-II photographic sky atlas. The survey covers the entire sky north of delta = -3 deg in 3 bands, calibrated to the Gunn $gri$ system, reaching to equivalent limiting magnitude of B_lim ~ 22 mag. As a result of the state-of-the-art digitisation of the plates, detailed processing of the scans, and a very extensive CCD calibration program, the data quality exceedes that of the previous photographically-based efforts. The end product of the survey will be the Palomar-Norris Sky Catalog, anticipated to contain > 50 million galaxies and > 2 billion stars, down to the survey classification limit, ~ 1 mag above the flux detection limit. Numerous scientific projects utilising these data have been started, and we describe briefly some of them; they illustrate the scientific potential of the data, and serve as the scientific verification tests of the survey. Finally, we discuss some general issues posed by the advent of multi-terabyte data sets in astronomy.

For faint photometric surveys our ability to quantify the clustering of galaxies has depended on interpreting the angular correlation function as a function of the limiting magnitude of the data. Due to the broad redshift distribution of galaxies at faint magnitude limits the correlation signal has been extremely difficult to detect and interpret. We introduce a new technique for measuring the evolution of clustering. We utilize photometric redshifts, derived from multicolor surveys, to isolate redshift intervals and calculate the evolution of the amplitude of the angular 2-pt correlation function. Applying these techniques to the the Hubble Deep Field we find that the shape of the correlation function, at z=1, is consistent with a power law with a slope of -0.8. For z>0.4 the best fit to the data is given by a model of clustering evolution with a comoving r0 = 2.37 Mpc and eps = -0.4 +/- 0.5, consistent with published measures of the clustering evolution. To match the canonical value of r0 = 5.4 Mpc, found for the clustering of local galaxies, requires a value of eps = 2.10 +/- 0.5 (significantly more than linear evolution). The log likelihood of this latter fit is 4.15 less than that for the r0 = 2.37 Mpc model. We, therefore, conclude that the parameterization of the clustering evolution of (1+z)^-(3+eps) is not a particularly good fit to the data.

Results of a blind test of photometric redshift predictions against spectroscopic galaxy redshifts obtained in the Hubble Deep Field with the Keck Telescope are presented. The best photometric redshift schemes predict spectroscopic redshifts with a redshift accuracy of |Delta-z|<0.1 for more than 68 percent of sources and with |Delta-z|<0.3 for 100 percent, when single-feature spectroscopic redshifts are removed from consideration. This test shows that photometric redshift schemes work well at least when the photometric data are of high quality and when the sources are at moderate redshifts.

The Hubble Deep Field (HDF) is the deepest set of multicolor optical photometric observations ever undertaken, and offers a valuable data set with which to study galaxy evolution. Combining the optical WFPC2 data with ground-based near-infrared photometry, we derive photometrically estimated redshifts for HDF galaxies with J<23.5. We demonstrate that incorporating the near-infrared data reduces the uncertainty in the estimated redshifts by approximately 40% and is required to remove systematic uncertainties within the redshift range 1<z<2. Utilizing these photometric redshifts, we determine the evolution of the comoving ultraviolet (2800 A) luminosity density (presumed to be proportional to the global star formation rate) from a redshift of z=0.5 to z=2. We find that the global star formation rate increases rapidly with redshift, rising by a factor of 12 from a redshift of zero to a peak at z~1.5. For redshifts beyond 1.5, it decreases monotonically. Our measures of the star formation rate are consistent with those found by Lilly et al. (1996) from the CFRS at z<1, and by Madau et al. (1996) from Lyman break galaxies at z > 2, and bridge the redshift gap between those two samples. The overall star formation or metal enrichment rate history is consistent with the predictions of Pei and Fall (1995) based on the evolving HI content of Lyman-alpha QSO absorption line systems.

We present the initial results from a deep, multi-band photometric survey of selected high Galactic latitude redshift fields. Previous work using the photographic data of Koo and Kron demonstrated that the distribution of galaxies in the multi-dimensional flux space U B R I is nearly planar. The position of a galaxy within this plane is determined by its redshift, luminosity and spectral type. Using recently acquired deep CCD photometry in existing, published redshift fields, we have redetermined the distribution of galaxies in this four-dimensional magnitude space. Furthermore, from our CCD photometry and the published redshifts, we have quantified the photometric-redshift relation within the standard AB magnitude system. This empirical relation has a measured dispersion of approximately 0.02 for z < 0.4. With this work we are reaching the asymptotic intrinsic dispersions that were predicted from simulated distributions of galaxy colors.

Based on a deep ROSAT/PSPC observation, we reveal an elongated complex of extended X-ray-emitting objects in and around the galaxy cluster A2125. Multicolor optical imaging of galaxies in the field suggests that this complex represents a hierarchical superstructure spanning about 11 Mpc at redshift z = 0.247. The multi-peak X-ray morphology of A2125 suggests that the cluster is an ongoing coalescence of at least three major subunits. The dynamical youth of this cluster is consistent with its large fraction of blue galaxies observed by Butcher & Oemler. The superstructure contains two additional clusters, projected at distances of only 3 and 4.3 Mpc from A2125. But the most interesting feature is the low-surface-brightness X-ray emission from a moderate galaxy concentration not associated with individual clusters. The emission likely arises in a hot intergalactic medium, as predicted in N-body/hydro simulations of structure formation.