Counts-in-cells (CIC) measurements contain a wealth of cosmological information yet are seldom used to constrain theories. Although we can predict the shape of the distribution for a given cosmology, to fit a model to the observed CIC probabilities requires the covariance matrix -- both the variance of counts in one probability bin and the covariance between counts in different bins. To date, there have been no general expressions for these variances. Here we show that correlations of particular levels, or "slices," of the density field determine the variance and covariance of CIC probabilities. We derive explicit formulae that accurately predict the variance and covariance among subvolumes of a simulated galaxy catalog, opening the door to the use of CIC measurements for cosmological parameter estimation.

The classical gravitational force on a torus is anisotropic and always lower than Newton's $1/r^2$ law. We demonstrate the effects of periodicity in dark matter only $N$-body simulations of spherical collapse and standard $\Lambda$CDM initial conditions. Periodic boundary conditions cause an overall negative and anisotropic bias in cosmological simulations of cosmic structure formation. The effect is most significant when the largest mildly non-linear scales are comparable to the linear size of the simulation box, as often is the case for high-resolution hydrodynamical simulations. Spherical collapse morphs into a shape similar to an octahedron. The anisotropic growth distorts the large-scale $\Lambda$CDM dark matter structures. We introduce the direction-dependent power spectrum invariant under the octahedral group of the simulation volume and show that the results break spherical symmetry.

The counts-in-cells (CIC) galaxy probability distribution depends on both the dark matter clustering amplitude $\sigma_8$ and the galaxy bias $b$. We present a theory for the CIC distribution based on a previous prescription of the underlying dark matter distribution and a linear volume transformation to redshift space. We show that, unlike the power spectrum, the CIC distribution breaks the degeneracy between $\sigma_8$ and $b$ on scales large enough that both bias and redshift distortions are still linear; thus we obtain a simultaneous fit for both parameters. We first validate the technique on the Millennium Simulation and then apply it to the SDSS Main Galaxy Sample. We find $\sigma_8 = 0.94^{+.11}_{-.10}$ and $b = 1.36^{+.14}_{-.11}$, consistent with previous complementary results from redshift distortions and from Planck.

The standard cosmological paradigm narrates a reassuring story of a universe currently dominated by an enigmatic dark energy component. Disquietingly, its universal explaining power has recently been challenged by, above all, the $\sim4\sigma$ tension in the values of the Hubble constant. A more durable yet less studied other anomaly is the repeated observation of integrated Sachs-Wolfe imprints $\sim5\times$ stronger than expected in the $\Lambda$CDM model from $R_v > 100\,h^{-1}Mpc$ super-structures. Here we show that the inhomogeneous AvERA model of emerging curvature is capable of telling a plausible albeit radically different story that explains both observational anomalies without dark energy. We demonstrate that while stacked imprints of $R_v > 100\,h^{-1}Mpc$ supervoids in cosmic microwave background temperature maps can discriminate between the AvERA and $\Lambda$CDM models, their characteristic differences may remain hidden using alternative void definitions and stacking methodologies. Testing the extremes, we then also show that the CMB Cold Spot can plausibly be explained in the AvERA model as an ISW imprint. We hence conclude that the anomalous imprint of supervoids may well be the canary in the coal mine, and existing observational evidence for dark energy should be re-interpreted to further test alternative models.

We present a novel test of general relativity (GR): measuring the geometric component of the time delay due to gravitational lensing. GR predicts that photons and gravitational waves follow the same geodesic paths and thus experience the same geometric time delay. We show that for typical systems, the time delays are tens of seconds, and thus can dominate over astrophysical delays in the timing of photon emission. For the case of GW 170817, we use a multi-plane lensing code to evaluate the time delay due to four massive halos along the line of sight. From literature mass and distance measurements of these halos, we establish at high confidence (significantly greater than 5 sigma) that the gravitational waves of GW 170817 underwent gravitational deflection to arrive within 1.7 seconds of the photons.

Repp and Szapudi (2019) present a physically-motivated galaxy bias model which remains physical in low-density regions and which also provides a better fit to simulation data than do typical survey-analysis bias models. Given plausible simplifying assumptions, the physics of this model (surprisingly) proves to be analogous to the Ising model of statistical mechanics. In the present work we present a method of testing this Ising bias model against empirical galaxy survey data. Using this method, we compare our model (as well as three reference models -- linear, quadratic, and logarithmic) to SDSS, 6dFGS, and COSMOS2015 results, finding that for spectroscopic redshift surveys, the Ising bias model provides a superior fit compared to the reference models. Photometric redshifts, on the other hand, introduce enough error into the radial coordinate that none of the models yields a good fit. A physically meaningful galaxy bias model is necessary for optimal extraction of cosmological information from dense galaxy surveys such as Euclid and WFIRST.

The Pan-STARRS1 (PS1) $3\pi$ survey is a comprehensive optical imaging survey of three quarters of the sky in the $grizy$ broad-band photometric filters. We present the methodology used in assembling the source classification and photometric redshift (photo-z) catalogue for PS1 $3\pi$ Data Release 1, titled Pan-STARRS1 Source Types and Redshifts with Machine learning (PS1-STRM). For both main data products, we use neural network architectures, trained on a compilation of public spectroscopic measurements that has been cross-matched with PS1 sources. We quantify the parameter space coverage of our training data set, and flag extrapolation using self-organizing maps. We perform a Monte-Carlo sampling of the photometry to estimate photo-z uncertainty. The final catalogue contains $2,902,054,648$ objects. On our validation data set, for non-extrapolated sources, we achieve an overall classification accuracy of $98.1\%$ for galaxies, $97.8\%$ for stars, and $96.6\%$ for quasars. Regarding the galaxy photo-z estimation, we attain an overall bias of $\left<\Delta z_{\mathrm{norm}}\right>=0.0005$, a standard deviation of $\sigma(\Delta z_{\mathrm{norm}})=0.0322$, a median absolute deviation of $\mathrm{MAD}(\Delta z_{\mathrm{norm}})=0.0161$, and an outlier fraction of $O=1.89\%$. The catalogue will be made available as a high-level science product via the Mikulski Archive for Space Telescopes at https://doi.org/10.17909//t9-rnk7-gr88.

We perform an analysis of two-point galaxy clustering and galaxy bias using Subaru Hyper-Suprime Cam (HSC) data taken jointly by the Subaru Strategic Program and the University of Hawaii in the COSMOS field. The depth of the data is similar to the ongoing Hawaii Two-0 (H20) optical galaxy survey, thus the results are indicative of future constraints from tenfold area. We measure the angular auto-power spectra of the galaxy overdensity in three redshift bins, defined by dropouts from the g-, r- and i-bands, and compare them to the theoretical expectation from concordance cosmology with linear galaxy bias. We determine the redshift distribution of each bin using a standard template-based photometric redshift method, coupled with a self-organizing map (SOM) to quantify colour space coverage. We also investigate sources of systematic errors to inform the methodology and requirements for Hawaii Two-0. The linear galaxy bias fit results are $b_{\mathrm{gal,g}} = 3.90 \pm 0.33 (\mathrm{stat}) \substack{ +0.64 \\ -0.24 } (\mathrm{sys})$ at redshift $z \simeq 3.7$, $b_{\mathrm{gal,r}} = 8.44 \pm 0.63 (\mathrm{stat}) \substack{ +1.42 \\ -0.72 } (\mathrm{sys})$ at $z \simeq 4.7$, and $b_{\mathrm{gal,i}} = 11.94 \pm 2.24 (\mathrm{stat}) \substack{ +1.82 \\ -1.27 } (\mathrm{sys})$ at $z \simeq 5.9$.

A reliable model of galaxy bias is necessary for interpreting data from future dense galaxy surveys. Conventional bias models are inaccurate, in that they can yield unphysical results ($\delta_g < -1$) for voids that might contain half of the available cosmological information. For this reason, we present a physically-motivated bias model based on an analogy with the Ising model. With only two free parameters, the model produces sensible results for both high- and low-density regions. We also test the model using a catalog of Millennium Simulation galaxies in cubical survey pixels with side lengths from $2h^{-1}$--$31h^{-1}$Mpc, at redshifts from 0 to 2. We find the Ising model markedly superior to linear and quadratic bias models on scales smaller than $10h^{-1}$Mpc, while those conventional models fare better on scales larger than $30h^{-1}$Mpc. While the largest scale where the Ising model is applicable might vary for a specific galaxy catalog, it should be superior on any scale with a non-negligible fraction of cells devoid of galaxies.

Future galaxy surveys hope to realize significantly tighter constraints on various cosmological parameters. The higher number densities achieved by these surveys will allow them to probe the smaller scales affected by non-linear clustering. However, in these regimes, the standard power spectrum can extract only a portion of such surveys' cosmological information. In contrast, the alternate statistic $A^*$ has the potential to double these surveys' information return, provided one can predict the $A^*$-power spectrum for a given cosmology. Thus, in this work we provide a prescription for this power spectrum $P_{A^*}(k)$, finding that the prescription is typically accurate to about 5 per cent for near-concordance cosmologies. This prescription will thus allow us to multiply the information gained from surveys such as Euclid and WFIRST.

Evaluation of gravitational theories by means of cosmological data suffers from the fact that galaxies are biased tracers of dark matter. Current bias models focus primarily on high-density regions, whereas low-density regions carry significant amounts of information relevant to the constraint of dark energy and alternative gravity theories. Thus, proper treatment of both high and low densities is important for future surveys. Accordingly, we here present an interactionless Ising model for this bias, and we demonstrate that it exhibits a remarkably good fit to both Millennium Simulation and Sloan Digital Sky Survey data, at both density extremes. The quality of the fit indicates that galaxy formation is (to first order) an essentially local process determined by initial conditions.

We present the multi-GPU realization of the StePS (Stereographically Projected Cosmological Simulations) algorithm with MPI-OpenMP-CUDA hybrid parallelization and nearly ideal scale-out to multiple compute nodes. Our new zoom-in cosmological direct N-body simulation method simulates the infinite universe with unprecedented dynamic range for a given amount of memory and, in contrast to traditional periodic simulations, its fundamental geometry and topology match observations. By using a spherical geometry instead of periodic boundary conditions, and gradually decreasing the mass resolution with radius, our code is capable of running simulations with a few gigaparsecs in diameter and with a mass resolution of $\sim 10^{9}M_{\odot}$ in the center in four days on three compute nodes with four GTX 1080Ti GPUs in each. The code can also be used to run extremely fast simulations with reasonable resolution for fitting cosmological parameters. These simulations are useful for prediction needs of large surveys. The StePS code is publicly available for the research community.

Cosmic voids, the less dense patches of the Universe, are promising laboratories to extract cosmological information. Thanks to their unique low density character, voids are extremely sensitive to diffuse components such as neutrinos and dark energy, and represent ideal environments to study modifications of gravity, where the effects of such modifications are expected to be more prominent. Robust void-related observables, including for example redshift-space distortions (RSD) and weak lensing around voids, are a promising way to chase and test new physics. Cosmological analysis of the large-scale structure of the Universe predominantly relies on the high density regions. Current and upcoming surveys are designed to optimize the extraction of cosmological information from these zones, but leave voids under-exploited. A dense, large area spectroscopic survey with imaging capabilities is ideal to exploit the power of voids fully. Besides helping illuminate the nature of dark energy, modified gravity, and neutrinos, this survey will give access to a detailed map of under-dense regions, providing an unprecedented opportunity to observe and study a so far under-explored galaxy population.

The distribution of matter in the universe is, to first order, lognormal. Improving this approximation requires characterization of the third moment (skewness) of the log density field. Thus, using Millennium Simulation phenomenology and building on previous work, we present analytic fits for the mean, variance, and skewness of the log density field $A$. We further show that a Generalized Extreme Value (GEV) distribution accurately models $A$; we submit that this GEV behavior is the result of strong intrapixel correlations, without which the smoothed distribution would tend (by the Central Limit Theorem) toward a Gaussian. Our GEV model yields cumulative distribution functions accurate to within 1.7 per cent for near-concordance cosmologies, over a range of redshifts and smoothing scales.

A primary goal of galaxy surveys is to tighten constraints on cosmological parameters, and the power spectrum $P(k)$ is the standard means of doing so. However, at translinear scales $P(k)$ is blind to much of these surveys' information---information which the log density power spectrum recovers. For discrete fields (such as the galaxy density), $A^*$ denotes the statistic analogous to the log density: $A^*$ is a "sufficient statistic" in that its power spectrum (and mean) capture virtually all of a discrete survey's information. However, the power spectrum of $A^*$ is biased with respect to the corresponding log spectrum for continuous fields, and to use $P_{A^*}(k)$ to constrain the values of cosmological parameters, we require some means of predicting this bias. Here we present a prescription for doing so; for Euclid-like surveys (with cubical cells 16$h^{-1}$ Mpc across) our bias prescription's error is less than 3 per cent. This prediction will facilitate optimal utilization of the information in future galaxy surveys.

The most common statistic used to analyze large-scale structure surveys is the correlation function, or power spectrum. Here, we show how `slicing' the correlation function on local density brings sensitivity to interesting non-Gaussian features in the large-scale structure, such as the expansion or contraction of baryon acoustic oscillations (BAO) according to the local density. The sliced correlation function measures the large-scale flows that smear out the BAO, instead of just correcting them as reconstruction algorithms do. Thus, we expect the sliced correlation function to be useful in constraining the growth factor, and modified gravity theories that involve the local density. Out of the studied cases, we find that the run of the BAO peak location with density is best revealed when slicing on a $\sim 40$ Mpc/$h$ filtered density. But slicing on a $\sim100$ Mpc/$h$ filtered density may be most useful in distinguishing between underdense and overdense regions, whose BAO peaks are separated by a substantial $\sim 5$ Mpc/$h$ at $z=0$. We also introduce `curtain plots' showing how local densities drive particle motions toward or away from each other over the course of an $N$-body simulation.

We present a novel $N$-body simulation method that compactifies the infinite spatial extent of the Universe into a finite sphere with isotropic boundary conditions to follow the evolution of the large-scale structure. Our approach eliminates the need for periodic boundary conditions, a mere numerical convenience which is not supported by observation and which modifies the law of force on large scales in an unrealistic fashion. We demonstrate that our method outclasses standard simulations executed on workstation-scale hardware in dynamic range, it is balanced in following a comparable number of high and low $k$ modes and, its fundamental geometry and topology match observations. Our approach is also capable of simulating an expanding, infinite universe in static coordinates with Newtonian dynamics. The price of these achievements is that most of the simulated volume has smoothly varying mass and spatial resolution, an approximation that carries different systematics than periodic simulations. Our initial implementation of the method is called StePS which stands for Stereographically Projected Cosmological Simulations. It uses stereographic projection for space compactification and naive $\mathcal{O}(N^2)$ force calculation which is nevertheless faster to arrive at a correlation function of the same quality than any standard (tree or P$^3$M) algorithm with similar spatial and mass resolution. The $N^2$ force calculation is easy to adapt to modern graphics cards, hence our code can function as a high-speed prediction tool for modern large-scale surveys. To learn about the limits of the respective methods, we compare StePS with GADGET-2 \citep{Gadget2_2005MNRAS.364.1105S} running matching initial conditions.

The recent AvERA cosmological simulation of R\'acz et al. (2017) has a $\Lambda \mathrm{CDM}$-like expansion history and removes the tension between local and Planck (cosmic microwave background) Hubble constants. We contrast the AvERA prediction of the integrated Sachs--Wolfe (ISW) effect with that of $\Lambda \mathrm{CDM}$. The linear ISW effect is proportional to the derivative of the growth function, thus it is sensitive to small differences in the expansion histories of the respective models. We create simulated ISW maps tracing the path of light-rays through the Millennium XXL cosmological simulation, and perform theoretical calculations of the ISW power spectrum. AvERA predicts a significantly higher ISW effect than $\Lambda \mathrm{CDM}$, $A=1.93-5.29$ times larger depending on the $l$ index of the spherical power spectrum, which could be utilized to definitively differentiate the models. We also show that AvERA predicts an opposite-sign ISW effect in the redshift range $z \approx 1.5 - 4.4$, in clear contrast with $\Lambda \mathrm{CDM}$. Finally, we compare our ISW predictions with previous observations. While at present these cannot distinguish between the two models due to large error bars, and lack of internal consistency suggesting systematics, ISW probes from future surveys will tightly constrain the models.

The cosmological dark matter field is not completely described by its hierarchy of $N$-point functions, a non-perturbative effect with the consequence that only part of the theory can be probed with the hierarchy. We give here an exact characterization of the joint information of the full set of $N$-point correlators of the lognormal field. The lognormal field is the archetypal example of a field where this effect occurs, and, at the same time, one of the few tractable and insightful available models to specify fully the statistical properties of the evolved matter density field beyond the perturbative regime. Nonlinear growth in the Universe in that model is set letting the log-density field probability density functional evolve keeping its Gaussian shape, according to the diffusion equation in Euclidean space. We show that the hierarchy probes a different evolution equation, the diffusion equation defined not in Euclidean space but on the compact torus, with uniformity as the long-term solution. The extraction of the hierarchy of correlators can be recast in the form of a nonlinear transformation applied to the field, 'wrapping', undergoing a sharp transition towards complete disorder in the deeply nonlinear regime, where all memory of the initial conditions is lost.

We report the results of the 2dF-VST ATLAS Cold Spot galaxy redshift survey (2CSz) based on imaging from VST ATLAS and spectroscopy from 2dF AAOmega over the core of the CMB Cold Spot. We sparsely surveyed the inner 5$^{\circ}$ radius of the Cold Spot to a limit of $i_{AB} \le 19.2$, sampling $\sim7000$ galaxies at $z<0.4$. We have found voids at $z=$ 0.14, 0.26 and 0.30 but they are interspersed with small over-densities and the scale of these voids is insufficient to explain the Cold Spot through the $\Lambda$CDM ISW effect. Combining with previous data out to $z\sim1$, we conclude that the CMB Cold Spot could not have been imprinted by a void confined to the inner core of the Cold Spot. Additionally we find that our 'control' field GAMA G23 shows a similarity in its galaxy redshift distribution to the Cold Spot. Since the GAMA G23 line-of-sight shows no evidence of a CMB temperature decrement we conclude that the Cold Spot may have a primordial origin rather than being due to line-of-sight effects.

According to the separate universe conjecture, spherically symmetric sub-regions in an isotropic universe behave like mini-universes with their own cosmological parameters. This is an excellent approximation in both Newtonian and general relativistic theories. We estimate local expansion rates for a large number of such regions, and use a scale parameter calculated from the volume-averaged increments of local scale parameters at each time step in an otherwise standard cosmological $N$-body simulation. The particle mass, corresponding to a coarse graining scale, is an adjustable parameter. This mean field approximation neglects tidal forces and boundary effects, but it is the first step towards a non-perturbative statistical estimation of the effect of non-linear evolution of structure on the expansion rate. Using our algorithm, a simulation with an initial $\Omega_m=1$ Einstein--de~Sitter setting closely tracks the expansion and structure growth history of the $\Lambda$CDM cosmology. Due to small but characteristic differences, our model can be distinguished from the $\Lambda$CDM model by future precision observations. Moreover, our model can resolve the emerging tension between local Hubble constant measurements and the Planck best-fitting cosmology. Further improvements to the simulation are necessary to investigate light propagation and confirm full consistency with cosmic microwave background observations.

At translinear scales, the log power spectrum captures significantly more cosmological information than the standard power spectrum. At high wavenumbers $k$, the Fisher information in the standard power spectrum $P(k)$ fails to increase in proportion to $k$ in part due to correlations between large- and small-scale modes. As a result, $P(k)$ suffers from an information plateau on these translinear scales, so that analysis with the standard power spectrum cannot access the information contained in these small-scale modes. The log power spectrum $P_A(k)$, on the other hand, captures the majority of this otherwise lost information. Until now there has been no means of predicting the amplitude of the log power spectrum apart from cataloging the results of simulations. We here present a cosmology-independent prescription for the log power spectrum; this prescription displays accuracy comparable to that of Smith et al. (2003), over a range of redshifts and smoothing scales, and for wavenumbers up to $1.5h$ Mpc$^{-1}$.

We have searched for the signature of cosmic voids in the CMB, in both the Planck temperature and lensing-convergence maps; voids should give decrements in both. We use zobov voids from the DR12 SDSS CMASS galaxy sample. We base our analysis on N-body simulations, to avoid a posteriori bias. For the first time, we detect the signature of voids in CMB lensing: the significance is $3.2\sigma$, close to $\Lambda$CDM in both amplitude and projected density-profile shape. A temperature dip is also seen, at modest significance ($2.3\sigma$), with amplitude about 6 times the prediction. This temperature signal is induced mostly by voids with radius between 100 and 150 Mpc/h, while the lensing signal is mostly contributed by smaller voids -- as expected; lensing relates directly to density, while ISW depends on gravitational potential. The void abundance in observations and simulations agree, as well. We also repeated the analysis excluding lower-significance voids: no lensing signal is detected, with an upper limit of about twice the $\Lambda$CDM prediction. But the mean temperature decrement now becomes non-zero at the $3.7\sigma$ level (similar to that found by Granett et al.), with amplitude about 20 times the prediction. However, the observed dependence of temperature on void size is in poor agreement with simulations, whereas the lensing results are consistent with $\Lambda$CDM theory. Thus, the overall tension between theory and observations does not favour non-standard theories of gravity, despite the hints of an enhanced amplitude for the ISW effect from voids.

The Rapid Transient Surveyor (RTS) is a proposed rapid-response, high-cadence adaptive optics (AO) facility for the UH 2.2-m telescope on Maunakea. RTS will uniquely address the need for high-acuity and sensitive near-infrared spectral follow-up observations of tens of thousands of objects in mere months by combining an excellent observing site, unmatched robotic observational efficiency, and an AO system that significantly increases both sensitivity and spatial resolving power. We will initially use RTS to obtain the infrared spectra of ~4,000 Type Ia supernovae identified by the Asteroid Terrestrial-Impact Last Alert System over a two year period that will be crucial to precisely measuring distances and mapping the distribution of dark matter in the z < 0.1 universe. RTS will comprise an upgraded version of the Robo-AO laser AO system and will respond quickly to target-of-opportunity events, minimizing the time between discovery and characterization. RTS will acquire simultaneous-multicolor images with an acuity of 0.07-0.10" across the entire visible spectrum (20% i'-band Strehl in median conditions) and <0.16" in the near infrared, and will detect companions at 0.5" at contrast ratio of ~500. The system will include a high-efficiency prism integral field unit spectrograph: R = 70-140 over a total bandpass of 840-1830 nm with an 8.7" by 6.0" field of view (0.15" spaxels). The AO correction boosts the infrared point-source sensitivity of the spectrograph against the sky background by a factor of seven for faint targets, giving the UH 2.2-m the H-band sensitivity of a 5.7-m telescope without AO.

The primary science driver for 3D galaxy surveys is their potential to constrain cosmological parameters. Forecasts of these surveys' effectiveness typically assume Gaussian statistics for the underlying matter density, despite the fact that the actual distribution is decidedly non-Gaussian. To quantify the effect of this assumption, we employ an analytic expression for the power spectrum covariance matrix to calculate the Fisher information for BAO-type model surveys. We find that for typical number densities, at $k_\mathrm{max} = 0.5 h$ Mpc$^{-1}$, Gaussian assumptions significantly overestimate the information on all parameters considered, in some cases by up to an order of magnitude. However, after marginalizing over a six-parameter set, the form of the covariance matrix (dictated by $N$-body simulations) causes the majority of the effect to shift to the "amplitude-like" parameters, leaving the others virtually unaffected. We find that Gaussian assumptions at such wavenumbers can underestimate the dark energy parameter errors by well over 50 per cent, producing dark energy figures of merit almost 3 times too large. Thus, for 3D galaxy surveys probing the non-linear regime, proper consideration of non-Gaussian effects is essential.

The Cold Spot (CS) is a clear feature in the Cosmic Microwave Background (CMB); it could be of primordial origin, or caused by a intervening structure along the line of sight. We identified a large projected underdensity in the recently constructed WISE-2MASS all-sky infrared galaxy catalogue aligned with the Cold Spot direction at $(l,b)\approx(209^\circ,-57^\circ)$. It has an angular size of tens of degrees, and shows a $\sim20\%$ galaxy underdensity in the center. Moreover, we find another large underdensity in the projected WISE-2MASS galaxy map at $(l,b)\approx(101^\circ,46^\circ)$ (hereafter Draco Supervoid), also aligned with a CMB decrement, although less significant than that of the CS direction. Motivated by these findings, we develop spherically symmetric Lemaitre-Tolman-Bondi (LTB) compensated void models to explain the observed CMB decrements with these two underdensities, or "supervoids". Within our perturbative treatment of the LTB voids, we find that the Integrated Sachs-Wolfe and Riess-Sciama effects due to the Draco Supervoid can account for the CMB decrement observed in the same direction. On the contrary, the extremely deep CMB decrement in the CS direction is more difficult to explain by the presence of the CS supervoid only. Nevertheless, the probability of a random alignment between the CS and the corresponding supervoid is disfavored, and thus its contribution as a secondary anisotropy cannot be neglected. We comment on how the approximations used in this paper, in particular the assumption of spherical symmetry, could change quantitatively our conclusions and might provide a better explanation for the CMB CS.

We discuss an analytical approximation for the matter power spectrum covariance matrix and its inverse on translinear scales, $k \sim 0.1h - 0.8h/\textrm{Mpc}$ at $z = 0$. We proceed to give an analytical expression for the Fisher information matrix of the nonlinear density field spectrum, and derive implications for its cosmological information content. We find that the spectrum information is characterized by a pair of upper bounds, 'plateaux', caused by the trispectrum, and a 'knee' in the presence of white noise. The effective number of Fourier modes, normally growing as a power law, is bounded from above by these plateaux, explaining naturally earlier findings from $N$-body simulations. These plateaux limit best possible measurements of the nonlinear power at the percent level in a $h^{-3}\textrm{Gpc}^3$ volume; the extraction of model parameters from the spectrum is limited explicitly by their degeneracy to the nonlinear amplitude. The value of the first, super-survey (SS) plateau depends on the characteristic survey volume and the large scale power; the second, intra-survey (IS) plateau is set by the small scale power. While both have simple interpretations within the hierarchical \textit{Ansatz}, the SS plateau can be predicted and generalized to still smaller scales within Takada and Hu's spectrum response formalism. Finally, the noise knee is naturally set by the density of tracers.

FastSound is a galaxy redshift survey using the near-infrared Fiber Multi-Object Spectrograph (FMOS) mounted on the Subaru Telescope, targeting H$\alpha$ emitters at $z \sim 1.18$--$1.54$ down to the sensitivity limit of H$\alpha$ flux $\sim 2 \times 10^{-16} \ \rm erg \ cm^{-2} s^{-1}$. The primary goal of the survey is to detect redshift space distortions (RSD), to test General Relativity by measuring the growth rate of large scale structure and to constrain modified gravity models for the origin of the accelerated expansion of the universe. The target galaxies were selected based on photometric redshifts and H$\alpha$ flux estimates calculated by fitting spectral energy distribution (SED) models to the five optical magnitudes of the Canada France Hawaii Telescope Legacy Survey (CFHTLS) Wide catalog. The survey started in March 2012, and all the observations were completed in July 2014. In total, we achieved $121$ pointings of FMOS (each pointing has a $30$ arcmin diameter circular footprint) covering $20.6$ deg$^2$ by tiling the four fields of the CFHTLS Wide in a hexagonal pattern. Emission lines were detected from $\sim 4,000$ star forming galaxies by an automatic line detection algorithm applied to 2D spectral images. This is the first in a series of papers based on FastSound data, and we describe the details of the survey design, target selection, observations, data reduction, and emission line detections.

We use the "Dark Energy and Massive Neutrino Universe" (DEMNUni) simulations to compare the constraining power of "sufficient statistics" with the standard matter power spectrum on the sum of neutrino masses, $M_\nu \equiv \sum m_\nu$. In general, the power spectrum, even supplemented with higher moments of the distribution, captures only a fraction of the available cosmological information due to correlations between the Fourier modes. In contrast, the non-linear transform of sufficient statistics, approximated by a logarithmic mapping A=ln(1+\delta), was designed to capture all the available cosmological information contained in the matter clustering; in this sense it is an optimal observable. Our analysis takes advantage of the recent analytical model developed by Carron et al. 2014 to estimate both the matter power spectrum and the A-power spectrum covariance matrices. Using a Fisher information approach, we find that using sufficient statistics increases up to 8 times the available information on the total neutrino mass at z=0, thus tightening the constraints by almost a factor of 3 compared to the matter power spectrum.

Beyond the linear regime of structure formation, part of cosmological information encoded in galaxy clustering becomes inaccessible to the usual power spectrum. "Sufficient statistics", A*, were introduced recently to recapture the lost, and ultimately extract all, cosmological information. We present analytical approximations for the A* and traditional power spectra as well as for their covariance matrices in order to calculate analytically their cosmological information content in the context of Fisher information theory. Our approach allows the precise quantitative comparison of the techniques with each other and to the total information in the data, and provides insights into sufficient statistics. In particular, we find that while the A* power spectrum has a similar shape to the usual galaxy power spectrum, its amplitude is strongly modulated by small scale statistics. This effect is mostly responsible for the ability of the A* power spectrum to recapture the information lost for the usual power spectrum. We use our framework to forecast the best achievable cosmological constraints for projected surveys as a function of their galaxy density, and compare the information content of the two power spectra. We find that sufficient statistics extract all cosmological information, resulting in an approximately factor of ~2 gain for dense projected surveys at low redshift. This increase in the effective volume of projected surveys is consistent with previous numerical calculations.

We use the WISE-2MASS infrared galaxy catalog matched with Pan-STARRS1 (PS1) galaxies to search for a supervoid in the direction of the Cosmic Microwave Background Cold Spot. Our imaging catalog has median redshift $z\simeq 0.14$, and we obtain photometric redshifts from PS1 optical colours to create a tomographic map of the galaxy distribution. The radial profile centred on the Cold Spot shows a large low density region, extending over 10's of degrees. Motivated by previous Cosmic Microwave Background results, we test for underdensities within two angular radii, $5^\circ$, and $15^\circ$. The counts in photometric redshift bins show significantly low densities at high detection significance, $\gtrsim 5 \sigma$ and $\gtrsim 6 \sigma$, respectively, for the two fiducial radii. The line-of-sight position of the deepest region of the void is $z\simeq 0.15-0.25$. Our data, combined with an earlier measurement by Granett et al. 2010, are consistent with a large $R_{\rm void}=(220 \pm 50) h^{-1}Mpc $ supervoid with $\delta_{m} \simeq -0.14 \pm 0.04$ centered at $z=0.22\pm0.03$. Such a supervoid, constituting at least a $\simeq 3.3\sigma$ fluctuation in a Gaussian distribution of the $\Lambda CDM$ model, is a plausible cause for the Cold Spot.

We combine photometric information of the WISE and 2MASS all-sky infrared databases, and demonstrate how to produce clean and complete galaxy catalogs for future analyses. Adding 2MASS colors to WISE photometry improves star-galaxy separation efficiency substantially at the expense of loosing a small fraction of the galaxies. We find that 93% of the WISE objects within W1<15.2 mag have a 2MASS match, and that a class of supervised machine learning algorithms, Support Vector Machines (SVM), are efficient classifiers of objects in our multicolor data set. We constructed a training set from the SDSS PhotoObj table with known star-galaxy separation, and determined redshift distribution of our sample from the GAMA spectroscopic survey. Varying the combination of photometric parameters input into our algorithm we show that W1 - J is a simple and effective star-galaxy separator, capable of producing results comparable to the multi-dimensional SVM classification. We present a detailed description of our star-galaxy separation methods, and characterize the robustness of our tools in terms of contamination, completeness, and accuracy. We explore systematics of the full sky WISE-2MASS galaxy map, such as contamination from Moon glow. We show that the homogeneity of the full sky galaxy map is improved by an additional J<16.5 mag flux limit. The all-sky galaxy catalog we present in this paper covers 21,200 sq. degrees with dusty regions masked out, and has an estimated stellar contamination of 1.2% and completeness of 70.1% among 2.4 million galaxies with $z_{med}= 0.14$. WISE-2MASS galaxy maps with well controlled stellar contamination will be useful for spatial statistical analyses, including cross correlations with other cosmological random fields, such as the Cosmic Microwave Background. The same techniques also yield a statistically controlled sample of stars as well.

Owing to the mass-sheet degeneracy, cosmic shear maps do not probe directly the Fourier modes of the underlying mass distribution on scales comparable to the survey size and larger. To assess the corresponding effect on attainable cosmological parameter constraints, we quantify the information on super-survey modes in a lognormal model and, when interpreted as nuisance parameters, their degeneracies to cosmological parameters. Our analytical and numerical calculations clarify the central role of super-sample covariance (SSC) in shaping the statistical power of cosmological observables. Reconstructing the background modes from their non-Gaussian statistical dependence to small scales modes yields the renormalized convergence. This diagonalizes the spectrum covariance matrix, and the information content of the corresponding power spectrum is increased by a factor of two over standard methods. Unfortunately, careful calculation of the Cramer-Rao bound shows that the information recovery can never be made complete, any observable built from shear fields, including optimal sufficient statistics, are subject to severe information loss, typically $80\%$ to $90\%$ below $\ell \sim 3000$ for generic cosmological parameters. The lost information can only be recovered from additional, non-shear based data. Our predictions hold just as well for a tomographic analysis, and/or full sky surveys.

Beyond the linear regime, Fourier modes of cosmological random fields become correlated, and the power spectrum of density fluctuations contains only a fraction of the available cosmological information. To unveil this formerly hidden information, the A* non-linear transform was introduced; it is optimized both for the nonlinearities induced by gravity and observational noise. Quantifying the resulting increase of our knowledge of cosmological parameters, we forecast the constraints from the angular power spectrum and that of A* from l ~ 200 to 3000 for upcoming galaxy surveys such as: the Wide-Field Infrared Survey Telescope (WFIRST), the Large Synoptic Survey Telescope (LSST), Euclid, the Hyper Suprime-Cam (HSC) and the Dark Energy Survey (DES). We find that at low redshifts this new data analysis strategy can double the extracted information, effectively doubling the survey area. To test the accuracy of our forecasting and the power of our data analysis methods, we apply the A* transformation to the latest release of the Canada-France-Hawaii-Telescope Legacy Survey (CFHTLS) Wide. While this data set is too sparse to allow for more than modest gains (~1.1-1.2), the realized gain from our method is in excellent agreement with our forecast, thus verifying the robustness of our analysis and prediction pipelines.

The statistical translation invariance of cosmological random fields is broken by a finite survey boundary, correlating the observable Fourier modes. Standard methods for generating Gaussian fields either neglect these correlations, or are costly, or both. Here we report on a fast and exact simulation method applicable to a wide class of two-point statistics that requires the simulation of a periodic grid of only twice the survey side with fast Fourier transforms. Super-survey modes, dominating the covariance of power spectra beyond linear scales in galaxy surveys and causing the correlation of large and small scales, "beat coupling", or "super-sample" covariance, are precisely accounted for in non-linear transformations of the Gaussian field. As an application, we simulate the CFHTLS $\sim 7^\circ \times 7^\circ$ W1 galaxy density field, modeled as a Poisson sampling of a lognormal density field. We show that our simulations produce power spectra, $A^*$-power spectra, counts-in-cells probability distributions as well as covariances perfectly consistent with the data. In addition, our technique reproduces the information plateau beyond linear scales as observed previously in Sloan Digital Sky Survey galaxy catalogs and in $N$-body simulations. Our method is thus an efficient yet powerful simulation and prediction tool for galaxy survey data and covariances.

We use a WISE-2MASS-Pan-STARRS1 galaxy catalog to search for a supervoid in the direction of the Cosmic Microwave Background Cold Spot. We obtain photometric redshifts using our multicolor data set to create a tomographic map of the galaxy distribution. The radial density profile centred on the Cold Spot shows a large low density region, extending over 10's of degrees. Motivated by previous Cosmic Microwave Background results, we test for underdensities within two angular radii, $5^\circ$, and $15^\circ$. Our data, combined with an earlier measurement by Granett et al 2010, are consistent with a large $R_{\rm void}=(192 \pm 15)h^{-1} Mpc $ $(2\sigma)$ supervoid with $\delta \simeq -0.13 \pm 0.03$ centered at $z=0.22\pm0.01$. Such a supervoid, constituting a $\sim3.5 \sigma$ fluctuation in the $\Lambda CDM$ model, is a plausible cause for the Cold Spot.

The sufficient statistics of the one-point probability density function of the dark matter density field is worked out using cosmological perturbation theory and tested to the Millennium simulation density field. The logarithmic transformation is recovered for spectral index close to $-1$ as a special case of the family of power transformations. We then discuss how these transforms should be modified in the case of noisy tracers of the field and focus on the case of Poisson sampling. This gives us optimal local transformations to apply to galaxy survey data prior the extraction of the spectrum in order to capture most efficiently the information encoded in large scale structures.

We test the statistical isotropy of the universe by analyzing the distribution of WISE extragalactic sources that were also observed by 2MASS. We pay particular attention to color cuts and foreground marginalization in order to cull a uniform sample of extragalactic objects and avoid stars. We detect a dipole gradient in the number-counts with an amplitude of $\sim$0.05, somewhat larger than expectations based on local structures corresponding to the depth and (independently measured) bias of our WISE-2MASS sources. The direction of the dipole, $(l,b)\simeq (310\,^{\circ}, -15\,^{\circ})$, is in reasonably good agreement with that found previously in the (shallower) 2MASS Extended Source Catalog alone. Interestingly, the dipole direction is not far from the direction of the dipolar modulation in the CMB found by Planck, and also fairly closely matches large-scale-structure bulk-flow directions found by various groups using galaxies and type Ia supernovae. It is difficult, however, to draw specific conclusions from the near-agreement of these directions.

Standard inflationary hot big bang cosmology predicts small fluctuations in the Cosmic Microwave Background (CMB) with isotropic Gaussian statistics. All measurements support the standard theory, except for a few anomalies discovered in the Wilkinson Microwave Anisotropy Probe maps and confirmed recently by the Planck satellite. The Cold Spot is one of the most significant of such anomalies, and the leading explanation of it posits a large void that imprints this extremely cold area via the linear Integrated Sachs-Wolfe (ISW) effect due to the decay of gravitational potentials over cosmic time, or via the Rees-Sciama (RS) effect due to late-time non-linear evolution. Despite several observational campaigns targeting the Cold Spot region, to date no suitably large void was found at higher redshifts $z > 0.3$. Here we report the detection of an $R =(192 \pm 15) h^{-1}Mpc$ size supervoid of depth $\delta = -0.13 \pm 0.03$, and centred at redshift $z = 0.22$. This supervoid, possibly the largest ever found, is large enough to significantly affect the CMB via the non-linear RS effect, as shown in our Lemaitre-Tolman-Bondi framework. This discovery presents the first plausible explanation for any of the physical CMB anomalies, and raises the possibility that local large-scale structure could be responsible for other anomalies as well.

We measure the average temperature decrement on the cosmic microwave background (CMB) produced by voids selected in the SDSS DR7 spectroscopic redshift galaxy catalog, spanning redshifts $0<z<0.44$. We find an imprint of amplitude between 2.6 and 2.9$\mu K$ as viewed through a compensated top-hat filter scaled to the radius of each void; we assess the statistical significance of the imprint at ~2$\sigma$. We make crucial use of $N$-body simulations to calibrate our analysis. As expected, we find that large voids produce cold spots on the CMB through the Integrated Sachs-Wolfe (ISW) effect. However, we also find that small voids in the halo density field produce hot spots, because they reside in contracting, larger-scale overdense regions. This is an important effect to consider when stacking CMB imprints from voids of different radius. We have found that the same filter radius that gives the largest ISW signal in simulations also yields close to the largest detected signal in the observations. However, although it is low in significance, our measured signal is much higher-amplitude than expected from ISW in the concordance $\Lambda$CDM universe. The discrepancy is also at the ~2$\sigma$ level. We have demonstrated that our result is robust against the varying of thresholds over a wide range.

Beyond the linear regime, the power spectrum and higher order moments of the matter field no longer capture all cosmological information encoded in density fluctuations. While non-linear transforms have been proposed to extract this information lost to traditional methods, up to now, the way to generalize these techniques to discrete processes was unclear; ad hoc extensions had some success. We pointed out in Carron and Szapudi (2013) that the logarithmic transform approximates extremely well the optimal "sufficient statistics", observables that extract all information from the (continuous) matter field. Building on these results, we generalize optimal transforms to discrete galaxy fields. We focus our calculations on the Poisson sampling of an underlying lognormal density field. We solve and test the one-point case in detail, and sketch out the sufficient observables for the multi-point case. Moreover, we present an accurate approximation to the sufficient observables in terms of the mean and spectrum of a non-linearly transformed field. We find that the corresponding optimal non-linear transformation is directly related to the maximum a posteriori Bayesian reconstruction of the underlying continuous field with a lognormal prior as put forward in Kitaura et al (2010). Thus simple recipes for realizing the sufficient observables can be built on previously proposed algorithms that have been successfully implemented and tested in simulations.

The recent data release of ESA's Planck mission together with earlier WMAP releases provide the first opportunity to compare high resolution full sky Cosmic Microwave Background temperature anisotropy maps. To quantify the coherence of these maps beyond the power spectrum we introduce Generalized Phases, unit vectors in the (2l+1) dimensional representation spaces. For a Gaussian distribution, Generalized Phases are random and if there is non-Gaussianity, they represent most of the non-Gaussian information. The alignment of these unit vectors from two maps can be characterized by their angle, 0 deg expected for full coherence, and 90 deg for random vectors. We analyze maps from both missions with the same mask and Nside=512 resolution, and compare both power spectra and Generalized Phases. We find excellent agreement of the Generalize Phases of Planck Smica map with that of the WMAP Q,V,W maps, rejecting the null hypothesis of no correlations at 5 sigma for l's l<700, l<900 and l<1100, respectively, except perhaps for l<10. Using foreground reduced maps for WMAP increases the phase coherence. The observed coherence angles can be explained with a simple assumption of Gaussianity and a WMAP noise model neglecting Planck noise, except for low-intermediate l's there is a slight, but significant off-set, depending on WMAP band. On the same scales WMAP power spectrum is about 2.6% higher at a very high significance, while at higher l's there appears to be no significant bias. Using our theoretical tools, we predict the phase alignment of Planck with a hypothetical perfect noiseless CMB experiment, finding decoherence at l > 2900; below this value Planck can be used most efficiently to constrain non-Gaussianity.

We performed a comprehensive statistical analysis using complex phases of the a_lm coefficients computed from the most recent data of the Wilkinson Microwave Anisotropy Probe (WMAP). Our aim was to confirm or constrain the presence of non-Gaussianities in the data. We found phase correlations - that suggest non-Gaussianity - at high-l in a_lm coefficients by applying various statistical tests. Most of all, we detected a non-Gaussian signal reaching a significance of 4.7 sigma using random walk statistics and simulations. However, our conclusion is that the non-Gaussian behavior is due to contamination from galactic foregrounds that show up in small scales only. When masked out the contaminated regions, we found no significant non-Gaussianity. Furthermore, we constrained the f_NL parameter using CMB simulations that mimic primordial non-Gaussianity. Our estimate is f_NL=40 +/- 200, in agreement with previous measurements and inflationary expectations.

Recently, several studies proposed non-linear transformations, such as a logarithmic or Gaussianization transformation, as efficient tools to recapture information about the (Gaussian) initial conditions. During non-linear evolution, part of the cosmologically relevant information leaks out from the second moment of the distribution. This information is accessible only through complex higher order moments or, in the worst case, becomes inaccessible to the hierarchy. The focus of this work is to investigate these transformations in the framework of Fisher information using cosmological perturbation theory of the matter field with Gaussian initial conditions. We show that at each order in perturbation theory, there is a polynomial of corresponding order exhausting the information on a given parameter. This polynomial can be interpreted as the Taylor expansion of the maximally efficient "sufficient" observable in the non-linear regime. We determine explicitly this maximally efficient observable for local transformations. Remarkably, this optimal transform is essentially the simple power transform with an exponent related to the slope of the power spectrum; when this is -1, it is indistinguishable from the logarithmic transform. This transform Gaussianizes the distribution, and recovers the linear density contrast. Thus a direct connection is revealed between undoing of the non-linear dynamics and the efficient capture of Fisher information. Our analytical results were compared with measurements from the Millennium Simulation density field. We found that our transforms remain very close to optimal even in the deeply non-linear regime with \sigma^2 \sim 10.

We define and study statistical ensembles of matter density profiles describing spherically symmetric, virialized dark matter haloes of finite extent with a given mass and total gravitational potential energy. Our ensembles include spatial degrees of freedom only, a microstate being a spherically symmetric matter density function. We provide an exact solution for the grand canonical partition functional, and show its equivalence to that of the microcanonical ensemble. We obtain analytically the mean profiles that correspond to an overwhelming majority of microstates. All such profiles have an infinitely deep potential well, with the singular isothermal sphere arising in the infinite temperature limit. Systems with virial radius larger than gravitational radius exhibit a localization of a finite fraction of the energy in the very center. The universal logarithmic inner slope of unity of the NFW haloes is predicted at any mass and energy if an upper bound is set to the maximal depth of the potential well. In this case, the statistically favored mean profiles compare well to the NFW profiles. For very massive haloes the agreement becomes exact.

We measured the cross-correlation of the Wilkinson Microwave Anisotropy Probe (WMAP) 7 year temperature map and the full sky data release of the Wide-field Infrared Survey Explorer (WISE) galaxy map. Using careful mapmaking and masking techniques we find a positive cross-correlation signal. The results are fully consistent with a Lambda-CDM Universe, although not statistically significant. Our findings are robust against changing the galactic latitude cut from |b|>10 to |b|>20 and no color dependence was detected when we used WMAP Q, V or W maps. We confirm higher significance correlations found in the preliminary data release. The change in significance is consistent with cosmic variance.

Large-scale bulk flows in the Universe distort the initial density field, broadening the baryon-acoustic-oscillation (BAO) feature that was imprinted when baryons were strongly coupled to photons. Additionally, there is a small shift inward in the peak of the conventional overdensity correlation function, a mass-weighted statistic. This shift occurs when high density peaks move toward each other. We explore whether this shift can be removed by applying to the density field a transform (such as a logarithm) that gives fairer statistical weight to fluctuations in underdense regions. Using configuration-space perturbation theory in the Zel'dovich approximation, we find that the log-density correlation function shows a much smaller inward shift in the position of the BAO peak at low redshift than is seen in the overdensity correlation function. We also show that if the initial, Lagrangian density of matter parcels could be estimated at their Eulerian positions, giving a displaced-initial-density field, its peak shift would be even smaller. In fact, a transformed field that accentuates underdensities, such as the reciprocal of the density, pushes the peak the other way, outward. In our model, these shifts in the peak position can be attributed to shift terms, involving the derivative of the linear correlation function, that entirely vanish in this displaced-initial-density field.

We estimated the cross-power spectra of a galaxy sample from the Wide-field Infrared Survey Explorer (WISE) survey with the 7-year Wilkinson Microwave Anisotropy Probe (WMAP) temperature anisotropy maps. A conservatively-selected galaxy sample covers ~13000sq.deg, with a median redshift of z=0.15. Cross-power spectra show correlations between the two data sets with no discernible dependence on the WMAP Q, V and W frequency bands. We interpret these results in terms of the the Integrated Sachs-Wolfe (ISW) effect: for the |b|>20 deg sample at l=6-87, we measure the amplitude (normalized to be 1 for vanilla LambdaCDM expectation) of the signal to be 3.4+-1.1, i.e., 3.1 sigma detection. We discuss other possibilities, but at face value, the detection of the linear ISW effect in a flat universe is caused by large scale decaying potentials, a sign of accelerated expansion driven by Dark Energy.

We present a new approach to cosmological perturbations based on the theory of Lie groups and their representations. After re-deriving the standard covariant formalism from SO(3) considerations, we provide a new expansion of the perturbed Friedmann-Lemaitre-Robertson-Walker (FLRW) metric in terms of irreducible representations of the Lorentz group. The resulting decomposition splits into (scalar, scalar), (scalar, vector) and (vector, vector) terms. These equations directly correspond to the standard Lifshitz classification of cosmological perturbations using scalar, vector and tensor modes which arise from the irreducible SO(3) representation of the spatial part of the metric. While the Lorentz group basis matches the underlying local symmetries of the FLRW spacetime better than the SO(3), the new equations do not provide further simplification compared to the standard cosmological perturbation theory. We conjecture that this is due to the fact that the so(3,1) ~ su(2) x su(2) Lorentz algebra has no pair of commuting generators commuting with any of the translation group generators.

We present an algorithm for quickly generating multiple realizations of N-body simulations to be used, for example, for cosmological parameter estimation from surveys of large-scale structure. Our algorithm uses a new method to resample the large-scale (Gaussian-distributed) Fourier modes in a periodic N-body simulation box in a manner that properly accounts for the nonlinear mode-coupling between large and small scales. We find that our method for adding new large-scale mode realizations recovers the nonlinear power spectrum to sub-percent accuracy on scales larger than about half the Nyquist frequency of the simulation box. Using 20 N-body simulations, we obtain a power spectrum covariance matrix estimate that matches the estimator in Takahashi et al. (2009) (from 5000 simulations) with < 20% errors in all matrix elements. Comparing the rates of convergence, we determine that our algorithm requires ~ 8 times fewer simulations to achieve a given error tolerance in estimates of the power spectrum covariance matrix. The degree of success of our algorithm indicates that we understand the main physical processes that give rise to the correlations in the matter power spectrum. Namely, the large-scale Fourier modes modulate both the degree of structure growth through the variation in the effective local matter density and also the spatial frequency of small-scale perturbations through large-scale displacements. We expect our algorithm to be useful for noise modeling when constraining cosmological parameters from weak lensing (cosmic shear) and galaxy surveys, rescaling summary statistics of N-body simulations for new cosmological parameter values, and any applications where the influence of Fourier modes larger than the simulation size must be accounted for.

The matter density field exhibits a nearly lognormal probability density distribution (PDF) after entering into the nonlinear regime. Recently, it has been shown that the shape of the power spectrum of a logarithmically transformed density field is very close to the linear density power spectrum, motivating an analytic study of it. In this paper, we develop cosmological perturbation theory for the power spectrum of this field. Our formalism is developed in the context of renormalized perturbation theory, which helps to regulate the convergence behavior of the perturbation series, and of the Taylor- series expansion we use of the logarithmic mapping. This approach allows us to handle the critical issue of density smoothing in a straightforward way. We also compare our perturbative results with simulation measurements.

We find that, even in the presence of discreteness noise, a Gaussianizing transform (producing a more-Gaussian one-point distribution) reduces nonlinearities in the power spectra of cosmological matter and galaxy density fields, in many cases drastically. Although Gaussianization does increase the effective shot noise, it also increases the power spectrum's fidelity to the linear power spectrum on scales where the shot noise is negligible. Gaussianizing also increases the Fisher information in the power spectrum in all cases and resolutions, although the gains are smaller in redshift space than in real space. We also find that the gain in cumulative Fisher information from Gaussianizing peaks at a particular grid resolution that depends on the sampling level.

The zeroth-order component of the cosine expansion of the projected three-point correlation function is proposed for clustering analysis of cosmic large scale structure. These functions are third order statistics but can be measured similarly to the projected two-point correlations. Numerical experiments with N-body simulations indicate that the advocated statistics are redshift distortion free within 10% in the non-linear regime on scales ~0.2-10Mpc/h. Halo model prediction of the zeroth-order component of the projected three-point correlation function agrees with simulations within ~10%. This lays the ground work for using these functions to perform joint analyses with the projected two-point correlation functions, exploring galaxy clustering properties in the framework of the halo model and relevant extensions.

We build a model for the density and integrated Sachs-Wolfe (ISW) profile of supervoid and supercluster structures. Our model assumes that fluctuations evolve linearly from an initial Gaussian random field. We find these assumptions capable of describing N-body simulations and simulated ISW maps remarkably well on large scales. We construct an ISW map based on locations of superstructures identified previously in the SDSS Luminous Red Galaxy sample. A matched filter analysis of the cosmic microwave background confirms a signal at the $3.2-\sigma$ confidence level and estimates the radius of the underlying structures to be $55 \pm 28 ~h^{-1}$Mpc. The amplitude of the signal, however, is $2-\sigma$ higher than $\Lambda$CDM predictions.

We measure the matter probability distribution function (PDF) via counts in cells in a volume limited subsample of the Sloan Digital Sky Survey Luminous Red Galaxy Catalog on scales from $30 h^{-1}$Mpc to $150 h^{-1}$Mpc and estimate the linear Integrated Sachs--Wolfe effect produced by supervoids and superclusters in the tail of the PDF. We characterize the PDF by the variance, $S_3$, and $S_4$, and study in simulations the systematic effects due to finite volume, survey shape and redshift distortion. We compare our measurement to the prediction of $\Lambda$CDM with linear bias and find a good agreement. We use the moments to approximate the tail of the PDF with analytic functions. A simple Gaussian model for the superstructures appears to be consistent with the claim by Granett et al. that density fluctuations on $100 h^{-1}$Mpc scales produce hot and cold spots with $\Delta T \approx 10\mu K$ on the cosmic microwave background.

We use the formalism of Szapudi(2004} to derive full explicit expressions for the linear two-point correlation function, including redshift space distortions and large angle effects. We take into account a non-perturbative geometric term in the Jacobian, which is still linear in terms of the dynamics. This term had been identified previously (Kaiser 1987,Hamilton and Culhane 1996), but has been neglected in all subsequent explicit calculations of the linear redshift space two-point correlation function. Our results represent a significant correction to previous explicit expressions and are in excellent agreement with our measurements in the Hubble Volume Simulation.

We study cell count moments up to fifth order of the distributions of haloes, of halo substructures as a proxy for galaxies, and of mass in the context of the halo model and compare theoretical predictions to the results of numerical simulations. On scales larger than the size of the largest cluster, we present a simple point cluster model in which results depend only on cluster-cluster correlations and on the distribution of the number of objects within a cluster, or cluster occupancy. The point cluster model leads to expressions for moments of galaxy counts in which the volume-averaged moments on large scales approach those of the halo distribution and on smaller scales exhibit hierarchical clustering with amplitudes $S_k$ determined by moments of the occupancy distribution. In this limit, the halo model predictions are purely combinatoric, and have no dependence on halo profile, concentration parameter, or potential asphericity. The full halo model introduces only two additional effects: on large scales, haloes of different mass have different clustering strengths, introducing relative bias parameters; and on the smallest scales, halo structure is resolved and details of the halo profile become important, introducing shape-dependent form factors. Because of differences between discrete and continuous statistics, the hierarchical amplitudes for galaxies and for mass behave differently on small scales even if galaxy number is exactly proportional to mass, a difference that is not necessarily well described in terms of bias.

The Cold Spot on the Cosmic Microwave Background could arise due to a supervoid at low redshift through the integrated Sachs-Wolfe effect. We imaged the region with MegaCam on the Canada-France-Hawai'i Telescope and present galaxy counts in photometric redshift bins. We rule out the existence of a 100Mpc radius spherical supervoid with underdensity delta=-0.3 at 0.5<z<0.9 at high significance. The data are consistent with an underdensity at low redshift, but the fluctuations are within the range of cosmic variance and the low density areas are not contiguous on the sky. Thus, we find no strong evidence for a supervoid. We cannot resolve voids smaller than 50Mpc radius; however, these can only make a minor contribution to the CMB temperature decrement.

We construct a map of the time derivative of the gravitational potential traced by SDSS Luminous Red Galaxies. The potential decays on large scales due to cosmic acceleration, leaving an imprint on cosmic microwave background (CMB) radiation through the integrated Sachs-Wolfe (ISW) effect. With a template fit, we directly measure this signature on the CMB at a 2-sigma confidence level. The measurement is consistent with the cross-correlation statistic, strengthening the claim that dark energy is indeed the cause of the correlation. This new approach potentially simplifies the cosmological interpretation. Our constructed linear ISW map shows no evidence for degree-scale cold and hot spots associated with supervoid and supercluster structures. This suggests that the linear ISW effect in a concordance Lambda-CDM cosmology is insufficient to explain the strong CMB imprints from these structures that we previously reported.

We find that nonlinearities in the dark-matter power spectrum are dramatically smaller if the density field first undergoes a logarithmic mapping. In the Millennium simulation, this procedure produces a power spectrum with a shape hardly departing from the linear power spectrum for k <~ 1 h/Mpc at all redshifts. Also, this procedure unveils pristine Fisher information on a range of scales reaching a factor of 2-3 smaller than in the standard power spectrum, yielding 10 times more cumulative signal-to-noise at z=0.

We seek to improve estimates of the power spectrum covariance matrix from a limited number of simulations by employing a novel statistical technique known as shrinkage estimation. The shrinkage technique optimally combines an empirical estimate of the covariance with a model (the target) to minimize the total mean squared error compared to the true underlying covariance. We test this technique on N-body simulations and evaluate its performance by estimating cosmological parameters. Using a simple diagonal target, we show that the shrinkage estimator significantly outperforms both the empirical covariance and the target individually when using a small number of simulations. We find that reducing noise in the covariance estimate is essential for properly estimating the values of cosmological parameters as well as their confidence intervals. We extend our method to the jackknife covariance estimator and again find significant improvement, though simulations give better results. Even for thousands of simulations we still find evidence that our method improves estimation of the covariance matrix. Because our method is simple, requires negligible additional numerical effort, and produces superior results, we always advocate shrinkage estimation for the covariance of the power spectrum and other large-scale structure measurements when purely theoretical modeling of the covariance is insufficient.

We measure hot and cold spots on the microwave background associated with supercluster and supervoid structures identified in the Sloan Digital Sky Survey Luminous Red Galaxy catalog. The structures give a compelling visual imprint, with a mean temperature deviation of 9.6 +/- 2.2 microK, i.e. above 4 sigma. We interpret this as a detection of the late-time Integrated Sachs-Wolfe (ISW) effect, in which cosmic acceleration from dark energy causes gravitational potentials to decay, heating or cooling photons passing through density crests or troughs. In a flat universe, the linear ISW effect is a direct signal of dark energy.

The observed apparent acceleration of the universe is usually attributed to negative pressure from a mysterious dark energy. This acceleration causes the gravitational potential to decay, heating or cooling photons travelling through crests or troughs of large-scale matter density fluctuations. This phenomenon, the late-time integrated Sachs-Wolfe (ISW) effect, has been detected, albeit at low significance, by cross-correlating various galaxy surveys with the Cosmic Microwave Background (CMB). Recently, the best evidence has come from the statistical combination of results from multiple correlated galaxy data sets. Here we show that vast structures identified in a galaxy survey project an image onto the CMB; stacking regions aligned with superclusters produces a hot spot, and supervoids, a cold spot. At over 4 sigma, this is the clearest evidence of the ISW effect to date. For the first time, our findings pin the effect to discrete structures. The ISW signal from supervoids and superclusters can be combined with other cosmological probes to constrain dark energy and cosmological parameters. In addition, our findings make it more plausible that the extreme Cold Spot and other anomalies in the CMB are caused by supervoids.

We investigate large-amplitude baryon acoustic oscillations (BAO's) in off-diagonal entries of cosmological power-spectrum covariance matrices. These covariance-matrix BAO's describe the increased attenuation of power-spectrum BAO's caused by upward fluctuations in large-scale power. We derive an analytic approximation to covariance-matrix entries in the BAO regime, and check the analytical predictions using N-body simulations. These BAO's look much stronger than the BAO's in the power spectrum, but seem detectable only at about a one-sigma level in gigaparsec-scale galaxy surveys. In estimating cosmological parameters using matter or galaxy power spectra, including the covariance-matrix BAO's can have a several-percent effect on error-bar widths for some parameters directly related to the BAO's, such as the baryon fraction. Also, we find that including the numerous galaxies in small haloes in a survey can reduce error bars in these cosmological parameters more than the simple reduction in shot noise might suggest.

Scale transformations have played an extremely successful role in studies of cosmological large-scale structure by relating the non-linear spectrum of cosmological density fluctuations to the linear primordial power at longer wavelengths. Here we generalize this approach to investigate the usefulness of scale transformations for nonlinear higher-order statistics, specifically the bispectrum. We find that the bispectrum predicted by perturbation theory at tree-level can be rescaled to match the results of full numerical simulations in the weakly and intermediately nonlinear regimes, especially at high redshifts, with an accuracy that is surprising given the simplicity of the procedure used. This discovery not only offers a simple practical way of calculating the matter bispectrum, but also suggests that scale transformations may yet yield even deeper insights into the physics of hierarchical clustering.

We investigate the cosmological Fisher information in the non-linear dark-matter power spectrum in the context of the halo model. We find that there is a plateau in information content on translinear scales which is generic to all cosmological parameters we tried. There is a rise in information on smaller scales, but we find that it is quite degenerate among different cosmological parameters (except, perhaps, the tilt). This suggests that it could be difficult to constrain cosmological parameters using the non-linear regime of the dark-matter power spectrum. We suggest ways to get around this problem, such as removing the largest haloes from consideration in survey analysis.

We measure the large-scale real-space power spectrum P(k) using luminous red galaxies (LRGs) in the Sloan Digital Sky Survey (SDSS) and use this measurement to sharpen constraints on cosmological parameters from the Wilkinson Microwave Anisotropy Probe (WMAP). We employ a matrix-based power spectrum estimation method using Pseudo-Karhunen-Loeve eigenmodes, producing uncorrelated minimum-variance measurements in 20 k-bands of both the clustering power and its anisotropy due to redshift-space distortions, with narrow and well-behaved window functions in the range 0.01h/Mpc < k < 0.2h/Mpc. Results from the LRG and main galaxy samples are consistent, with the former providing higher signal-to-noise. Our results are robust to omitting angular and radial density fluctuations and are consistent between different parts of the sky. They provide a striking confirmation of the predicted large-scale LCDM power spectrum. Combining only SDSS LRG and WMAP data places robust constraints on many cosmological parameters that complement prior analyses of multiple data sets. The LRGs provide independent cross-checks on Om and the baryon fraction in good agreement with WMAP. Within the context of flat LCDM models, our LRG measurements complement WMAP by sharpening the constraints on the matter density, the neutrino density and the tensor amplitude by about a factor of two, giving Omega_m=0.24+-0.02 (1 sigma), sum m_nu < 0.9 eV (95%) and r<0.3 (95%). Baryon oscillations are clearly detected and provide a robust measurement of the comoving distance to the median survey redshift z=0.35 independent of curvature and dark energy properties. Within the LCDM framework, our power spectrum measurement improves the evidence for spatial flatness, sharpening the curvature constraint Omega_tot=1.05+-0.05 from WMAP alone to Omega_tot=1.003+-0.010. Assuming Omega_tot=1, the equation of state parameter is constrained to w=-0.94+-0.09, indicating the potential for more ambitious future LRG measurements to provide precision tests of the nature of dark energy. All these constraints are essentially independent of scales k>0.1h/Mpc and associated nonlinear complications, yet agree well with more aggressive published analyses where nonlinear modeling is crucial.

We measure the 2-1 cumulant correlator power spectrum $C^{21}_l$, a degenerate bispectrum, from the second data release of the Wilkinson Microwave Anisotropy Probe (WMAP). Our high resolution measurements with SpICE span a large configuration space ($\simeq 168\times999$) corresponding to the possible cross-correlations of the maps recorded by the different differencing assemblies. We present a novel method to recover the eigenmodes of the correspondingly large Monte Carlo covariance matrix. We examine its eigenvalue spectrum and use random matrix theory to show that the off diagonal terms are dominated by noise. We minimize the $\chi^2$ to obtain constraints for the non-linear coupling parameter $f_{NL} = 22 \pm 52 (1\sigma)$.

Using the halo model, we investigate the cosmological Fisher information in the non-linear dark-matter power spectrum about the initial amplitude of linear power. We find that there is little information on `translinear' scales (where the one- and two-halo terms are both significant) beyond what is on linear scales, but that additional information is present on small scales, where the one-halo term dominates. This behavior agrees with the surprising results that Rimes & Hamilton (2005, 2006) found using N-body simulations. We argue that the translinear plateau in cumulative information arises largely from fluctuations in the numbers of large haloes in a finite volume. This implies that more information could be extracted on non-linear scales if the masses of the largest haloes in a survey are known.

[Abridged] We use mock catalogues based on the GALICS model (Hatton et al. 03) to explore the nature of galaxy clustering observed in the SDSS. We measure low and high order angular clustering statistic from these mock catalogues, after selecting galaxies the same way as for observations, and compare them directly to estimates from SDSS data. Note that we also present measurements of S3-S5 on the SDSS DR1. We find that our model is in general good agreement with observations in the scale/luminosity range where we can trust the predictions. This range is found to be limited (i) by the size of the dark matter simulation used -- which introduces finite volume effects at large scales -- and by the mass resolution of this simulation -- which introduces incompleteness at apparent magnitudes fainter than $r\sim 20$. We then focus on the small scale clustering properties of galaxies and investigate the behaviour of three different prescriptions for positioning galaxies within haloes of dark matter. We show that galaxies are poor tracers both of DM particles or DM sub-structures, within groups and clusters. Instead, SDSS data tells us that the distribution of galaxies lies somewhat in between these two populations. This confirms the general theoretical expectation from numerical simulations and semi-analytic modelling.

Based on the cosmic holographic conjecture of Fischler and Susskind, we point out that the average energy density of the universe is bound from above by its entropy limit. Since Friedmann's equation saturates this relation, the measured value of the cosmological energy density is completely natural in the framework of holographic thermodynamics: vacuum energy density fills the available quantum degrees of freedom allowed by the holographic bound. This is in strong contrast with traditional quantum field theories where, since no similar bound applies, the natural value of the vacuum energy is expected to be 123 orders of magnitude higher than the holographic value. Based on our simple calculation, holographic thermodynamics, and consequently any future holographic quantum (gravity) theory, resolves the vacuum energy puzzle.

We present measurements of the normalised redshift-space three-point correlation function (Q_z) of galaxies from the Sloan Digital Sky Survey (SDSS) main galaxy sample. We have applied our "npt" algorithm to both a volume-limited (36738 galaxies) and magnitude-limited sample (134741 galaxies) of SDSS galaxies, and find consistent results between the two samples, thus confirming the weak luminosity dependence of Q_z recently seen by other authors. We compare our results to other Q_z measurements in the literature and find it to be consistent within the full jack-knife error estimates. However, we find these errors are significantly increased by the presence of the ``Sloan Great Wall'' (at z ~ 0.08) within these two SDSS datasets, which changes the 3-point correlation function (3PCF) by 70% on large scales (s>=10h^-1 Mpc). If we exclude this supercluster, our observed Q_z is in better agreement with that obtained from the 2dFGRS by other authors, thus demonstrating the sensitivity of these higher-order correlation functions to large-scale structures in the Universe. This analysis highlights that the SDSS datasets used here are not ``fair samples'' of the Universe for the estimation of higher-order clustering statistics and larger volumes are required. We study the shape-dependence of Q_z(s,q,theta) as one expects this measurement to depend on scale if the large scale structure in the Universe has grown via gravitational instability from Gaussian initial conditions. On small scales (s <= 6h^-1 Mpc), we see some evidence for shape-dependence in Q_z, but at present our measurements are consistent with a constant within the errors (Q_z ~ 0.75 +/- 0.05). On scales >10h^-1 Mpc, we see considerable shape-dependence in Q_z.

We study the luminosity and color dependence of the galaxy 2-point correlation function in the Sloan Digital Sky Survey, starting from a sample of 200,000 galaxies over 2500 deg^2. We concentrate on the projected correlation function w(r_p), which is directly related to the real space \xi(r). The amplitude of w(r_p) grows continuously with luminosity, rising more steeply above the characteristic luminosity L_*. Redder galaxies exhibit a higher amplitude and steeper correlation function at all luminosities. The correlation amplitude of blue galaxies increases continuously with luminosity, but the luminosity dependence for red galaxies is less regular, with bright red galaxies more strongly clustered at large scales and faint red galaxies more strongly clustered at small scales. We interpret these results using halo occupation distribution (HOD) models assuming concordance cosmological parameters. For most samples, an HOD model with two adjustable parameters fits the w(r_p) data better than a power-law, explaining inflections at r_p ~ 1-3 Mpc/h as the transition between the 1-halo and 2-halo regimes of \xi(r). The implied minimum mass for a halo hosting a central galaxy above a luminosity threshold L grows as M_min ~ L at low luminosities and more steeply above L_*. The mass at which an average halo has one satellite galaxy brighter than L is M_1 ~ 23 M_min(L). These results imply a conditional luminosity function (at fixed halo mass) in which central galaxies lie far above a Schechter function extrapolation of the satellite population. HOD models nicely explain the joint luminosity-color dependence of w(r_p) in terms of the color fractions of central and satellite populations as a function of halo mass. The inferred HOD properties are in good qualitative agreement with theoretical predictions.

We present a new method to estimate three-point correlations in Cosmic Microwave Background maps. Our Fast Fourier Transform based implementation estimates three-point functions using all possible configurations (triangles) at a controlled resolution. The speed of the technique depends both on the resolution and the total number of pixels $N$. The resulting $N\log N$ scaling is substantially faster than naive methods with prohibitive $N^3$ scaling. As an initial application, we measure three-point correlation functions in the First Year Data Release of the Wilkinson Anisotropy Probe. We estimate 336 cross-correlations of any triplet of maps from the 8 differential assemblies, scanning altogether 2.6 million triangular configurations. Estimating covariances from Gaussian signal plus realistic noise simulations, we perform a null-hypothesis testing with regards to the Gaussianity of the Cosmic Microwave Background. Our main result is that at the three-point level WMAP is fully consistent with Gaussianity. To quantify the level of possible deviations, we introduce false discovery rate analysis, a novel statistical technique to analyze for three-point measurements. This confirms that the data are consistent with Gaussianity at better than 1-$\sigma$ level when jointly considering all configurations. We constrain a specific non-Gaussian model using the quadratic approximation of weak non-Gaussianities in terms of the $f_{NLT}$ parameter, for which we construct an estimator from the the three-point function. We find that using the skewness alone is more constraining than a heuristic suboptimal combination of all our results; our best estimate is $f_{NLT} = -110\pm 150$ assuming a $\Lambda$CDM concordance model.

We perform detailed comparison of the semi-analytic halo model predictions with measurements in numerical simulations of the two and three point correlation functions (3PCF), as well as power spectrum and bispectrum. We discuss the accuracy and self-consistency of the halo model description of gravitational clustering in the non-linear regime and constrain halo model parameters. We exploit the recently proposed multipole expansion of three point statistics that expresses rotation invariance in the most natural way. This not only offers technical advantages by reducing the integrals required for the halo model predictions, but amounts to a convenient way of compressing the information contained in the 3PCF. We find that, with an appropriate choice of the halo boundary and mass function cut-off, halo model predictions are in good agreement with the bispectrum measured in numerical simulations. However, the halo model predicts less than the observed configuration dependence of the 3PCF on ~ Mpc scales. This effect is mainly due to quadrupole moment deficit, possibly related to the assumption of spherical halo geometry. Our analysis shows that using its harmonic decomposition, the full configuration dependence of the 3PCF in the non-linear regime can be compressed into just a few numbers, the lowest multipoles. Moreover, these multipoles are closely related to the highest signal to noise eigenmodes of the 3PCF. Therefore this estimator may simplify future analyses aimed at constraining cosmological and halo model parameters from observational data.

We measure the monopole moment of the three-point correlation function on scales $1\mpc-70\mpc$ in the Two degree Field Galaxy Redshift Survey (2dFGRS). Volume limited samples are constructed using a series of integral magnitudes bins between $M = -18 ... -22$. Our measurements with a novel edge-corrected estimator represent most, if not all, three-point level monopole or angular averaged information in the catalogue. We fit a perturbative non-linear bias model to a joint data vector formed from the estimated two- and three-point correlation functions. Two different models are used: an analytic model based on Eulerian perturbation theory including bias and redshift distortions, and a phenomenological bias model based on the direct redshift space measurements in the large Virgo simulations. To interpret the clustering results, we perform a three parameter Gaussian maximum likelihood analysis. In the canonical $-21 \sim -20$ volume limited sample we find $\sigma_8 = 0.93^{+0.06}_{-0.2}$, $b = 1.04^{+0.23}_{-0.09}$, and $b_2 = -0.06^{+0.003}_{-0.001}$. Our estimate of $\sigma_8$, is robust across the different volume limited samples constructed. These results, based solely on the large scale clustering of galaxies, are in excellent agreement with previous analyses using the Wilkinson Anisotropy Probe: this is a spectacular success of the concordance model. We also present two-parameter fits for the bias parameters, which are in excellent agreement with previous findings of the bias evolution in the 2dFGRS.

We discuss the construction of a photometric redshift catalogue of Luminous Red Galaxies (LRGs) from the Sloan Digital Sky Survey (SDSS), emphasizing the principal steps necessary for constructing such a catalogue -- (i) photometrically selecting the sample, (ii) measuring photometric redshifts and their error distributions, (iii) and estimating the true redshift distribution. We compare two photometric redshift algorithms for these data and find that they give comparable results. Calibrating against the SDSS and SDSS-2dF spectroscopic surveys, we find that the photometric redshift accuracy is $\sigma \sim 0.03$ for redshifts less than 0.55 and worsens at higher redshift ($\sim 0.06$). These errors are caused by photometric scatter, as well as systematic errors in the templates, filter curves, and photometric zeropoints. We also parametrize the photometric redshift error distribution with a sum of Gaussians, and use this model to deconvolve the errors from the measured photometric redshift distribution to estimate the true redshift distribution. We pay special attention to the stability of this deconvolution, regularizing the method with a prior on the smoothness of the true redshift distribution. The methods we develop are applicable to general photometric redshift surveys.

In this lecture notes I concentrate on the classic and widely applicable characterization of higher order statistics by joint moments, a.k.a. higher order correlation functions, and directly related statistics. I put special emphasis on recent results, especially the connection of spatial statistics with symmetries.

We present two related techniques to measure the two-point correlation function and the power spectrum with edge correction in any spatial dimensions. The underlying algorithm uses fast Fourier transforms for calculating the two-point function with an heuristically weighted edge corrected estimator. Once the correlation function is measured, we estimate the power spectrum via numerical integration of the Hankel transform connecting the two. We introduce an efficient numerical technique based on Gauss-Bessel-quadrature and double exponential transformation. This, combined with our, or any similar, two-point function estimator leads to a novel edge corrected estimator for power spectra. The pair of techniques presented are the Euclidean analogs of those developed and widely used in cosmic microwave background research for spherical maps.

We report the first result of the clustering analysis of Sloan Digital Sky Survey (SDSS) quasars. We compute the two-point correlation function (2PCF) of SDSS quasars in redshift space at $8h^{-1}{\rm Mpc} < s < 500h^{-1}{\rm Mpc}$, with particular attention to its baryonic signature. Our sample consists of 19986 quasars extracted from the SDSS Data Release 4 (DR4). The redshift range of the sample is $0.72 \le z \le 2.24$ (the mean redshift is $\bar z = 1.46$) and the reddening-corrected $i$-band apparent magnitude range is $15.0 \le m_{i,{\rm rc}} \le 19.1$. Due to the relatively low number density of the quasar sample, the bump in the power spectrum due to the baryon density, $\Omega_{\rm b}$, is not clearly visible. The effect of the baryon density is, however, to distort the overall shape of the 2PCF.The degree of distortion makes it an interesting alternate measure of the baryonic signature. Assuming a scale-independent linear bias and the spatially flat universe, i.e., $\Omega_{\rm b} + \Omega_{\rm d} + \Omega_\Lambda =1$, where $\Omega_{\rm d}$ and $\Omega_\Lambda$ denote the density parameters of dark matter and the cosmological constant, we combine the observed quasar 2PCF and the predicted matter 2PCF to put constraints on $\Omega_{\rm b}$ and $\Omega_\Lambda$. Our result is fitted as $0.80- 2.8\Omega_{\rm b} < \Omega_\Lambda < 0.90 - 1.4\Omega_{\rm b}$ at the 2$\sigma$ confidence level, which is consistent with results from other cosmological observations such as WMAP. (abridged)

We present the large-scale correlation function measured from a spectroscopic sample of 46,748 luminous red galaxies from the Sloan Digital Sky Survey. The survey region covers 0.72 h^{-3} Gpc^3 over 3816 square degrees and 0.16<z<0.47, making it the best sample yet for the study of large-scale structure. We find a well-detected peak in the correlation function at 100h^{-1} Mpc separation that is an excellent match to the predicted shape and location of the imprint of the recombination-epoch acoustic oscillations on the low-redshift clustering of matter. This detection demonstrates the linear growth of structure by gravitational instability between z=1000 and the present and confirms a firm prediction of the standard cosmological theory. The acoustic peak provides a standard ruler by which we can measure the ratio of the distances to z=0.35 and z=1089 to 4% fractional accuracy and the absolute distance to z=0.35 to 5% accuracy. From the overall shape of the correlation function, we measure the matter density Omega_mh^2 to 8% and find agreement with the value from cosmic microwave background (CMB) anisotropies. Independent of the constraints provided by the CMB acoustic scale, we find Omega_m = 0.273 +- 0.025 + 0.123 (1+w_0) + 0.137 Omega_K. Including the CMB acoustic scale, we find that the spatial curvature is Omega_K=-0.010+-0.009 if the dark energy is a cosmological constant. More generally, our results provide a measurement of cosmological distance, and hence an argument for dark energy, based on a geometric method with the same simple physics as the microwave background anisotropies. The standard cosmological model convincingly passes these new and robust tests of its fundamental properties.

We measure the angular power spectrum of the WMAP first-year temperature anisotropy maps. We use SpICE (Spatially Inhomogeneous Correlation Estimator) to estimate Cl's for multipoles l=2-900 from all possible cross-correlation channels. Except for the map-making stage, our measurements provide an independent analysis of that by Hinshaw etal (2003). Despite the different methods used, there is virtually no difference between the two measurements for l < 700 ; the highest l's are still compatible within 1-sigma errors. We use a novel intra-bin variance method to constrain Cl errors in a model independent way. When applied to WMAP data, the intra-bin variance estimator yields diagonal errors 10% larger than those reported by the WMAP team for 100 < l < 450. This translates into a 2.4 sigma detection of systematics since no difference is expected between the SpICE and the WMAP team estimator window functions in this multipole range. With our measurement of the Cl's and errors, we get chi^2/d.o.f. = 1.042 for a best-fit LCDM model, which has a 14% probability, whereas the WMAP team obtained chi^2/d.o.f. = 1.066, which has a 5% probability. We assess the impact of our results on cosmological parameters using Markov Chain Monte Carlo simulations. From WMAP data alone, assuming spatially flat power law LCDM models, we obtain the reionization optical depth tau = 0.145 +/- 0.067, spectral index n_s = 0.99 +/- 0.04, Hubble constant h = 0.67 +/- 0.05, baryon density Omega_b h^2 = 0.0218 +/- 0.0014, cold dark matter density Omega_{cdm} h^2 = 0.122 +/- 0.018, and sigma_8 = 0.92 +/- 0.12, consistent with a reionization redshift z_{re} = 16 +/- 5 (68% CL).

We present measurements of the redshift--space three-point correlation function of galaxies in the Sloan Digital Sky Survey (SDSS). For the first time, we analyze the dependence of this statistic on galaxy morphology, color and luminosity. In order to control systematics due to selection effects, we used $r$--band, volume-limited samples of galaxies, constructed from the magnitude-limited SDSS data ($14.5<r<17.5$), and further divided the samples into two morphological types (early and late) or two color populations (red and blue). The three-point correlation function of SDSS galaxies follow the hierarchical relation well and the reduced three-point amplitudes in redshift--space are almost scale-independent ($Q_z=0.5\sim1.0$). In addition, their dependence on the morphology, color and luminosity is not statistically significant. Given the robust morphological, color and luminosity dependences of the two-point correlation function, this implies that galaxy biasing is complex on weakly non-linear to non-linear scales. We show that simple deterministic linear relation with the underlying mass could not explain our measurements on these scales.

Conditional cumulants form a set of unique statistics which represent a sensible compromise between N-point correlation functions and cumulants measured from moments of counts in cells. They share accurate edge corrected estimators with $N$-point correlation functions, yet, they are as straightforward to measure and interpret as counts in cells. The conditional cumulants have three equivalent views as i) degenerate N-point correlation functions ii) or integrated monopole moments of the bispectrum iii) they are closely related to neighbour counts. We compute the predictions of weakly non-linear perturbation theory for conditional cumulants and compare them with measurements in simulations, both in real and redshift space. We find excellent agreement between theory and simulations, especially on scales >~20Mpc. Due to their advantageous statistical properties and well understood dynamics, we propose conditional cumulants as tools for high precision cosmology. Potential applications include constraining bias and redshift distortions from galaxy redshift surveys

We present a fast method for estimating the cosmic microwave background polarization power spectra using unbiased estimates of heuristically-weighted correlation functions. This extends the O(N_pix^(3/2)) method of Szapudi et al. (2001) to polarized data. If the sky coverage allows the correlation functions to be estimated over the full range of angular separations, they can be inverted directly with integral transforms and clean separation of the electric (E) and magnetic (B) modes of polarization is obtained exactly in the mean. We assess the level of E-B mixing that arises from apodized integral transforms when the correlation function can only be estimated for a subset of angular scales, and show that it is significant for small-area observations. We introduce new estimators to deal with this case on the spherical sky that preserve E-B separation; their construction requires an additional integration of the correlation functions but the computational cost is negligible. We illustrate our methods with application to a large-area survey with parameters similar to PLANCK, and the small-area Background Imaging of Cosmic Extragalactic Polarization experiment. In both cases we show that the errors on the recovered power spectra are close to theoretical expectations.

Multipole expansion of spatial three-point statistics is introduced as a tool for investigating and displaying configuration dependence. The novel parametrization renders the relation between bi-spectrum and three-point correlation function especially transparent as a set of two-dimensional Hankel transforms. It is expected on theoretical grounds, that three-point statistics can be described accurately with only a few multipoles. In particular, we show that in the weakly non-linear regime, the multipoles of the reduced bispectrum, $Q_l$, are significant only up to quadrupole. Moreover, the non-linear bias in the weakly non-linear regime only affects the monopole order of these statistics. As a consequence, a simple, novel set of estimators can be constructed to constrain galaxy bias. In addition, the quadrupole to dipole ratio is independent of the bias, thus it becomes a novel diagnostic of the underlying theoretical assumptions: weakly non-linear gravity and perturbative local bias. To illustrate the use of our approach, we present predictions based on both power law, and CDM models. We show that the presently favoured SDSS-WMAP concordance model displays strong ``baryon bumps'' in the $Q_l$'s. Finally, we sketch out three practical techniques estimate these novel quantities: they amount to new, and for the first time edge corrected, estimators for the bispectrum.

We explore linear redshift distortions in wide angle surveys from the point of view of symmetries. We show that the redshift space two-point correlation function can be expanded into tripolar spherical harmonics of zero total angular momentum $S_{l_1 l_2 l_3}(\hat x_1, \hat x_2, \hat x)$. The coefficients of the expansion $B_{l_1 l_2 l_3}$ are analogous to the $C_l$'s of the angular power spectrum, and express the anisotropy of the redshift space correlation function. Moreover, only a handful of $B_{l_1 l_2 l_3}$ are non-zero: the resulting formulae reveal a hidden simplicity comparable to distant observer limit. The $B_{l_1 l_2 l_3}$ depend on spherical Bessel moments of the power spectrum and $f = \Omega^{0.6}/b$. In the plane parallel limit, the results of \cite{Kaiser1987} and \cite{Hamilton1993} are recovered. The general formalism is used to derive useful new expressions. We present a particularly simple trigonometric polynomial expansion, which is arguably the most compact expression of wide angle redshift distortions. These formulae are suitable to inversion due to the orthogonality of the basis functions. An alternative Legendre polynomial expansion was obtained as well. This can be shown to be equivalent to the results of \cite{SzalayEtal1998}. The simplicity of the underlying theory will admit similar calculations for higher order statistics as well.

We present estimates of cosmological parameters from the application of the Karhunen-Loeve transform to the analysis of the 3D power spectrum of density fluctuations using Sloan Digital Sky Survey galaxy redshifts. We use Omega_m*h and f_b = Omega_b/Omega_m to describe the shape of the power spectrum, sigma8 for the (linearly extrapolated) normalization, and beta to parametrize linear theory redshift space distortions. On scales k < 0.16 h/Mpc, our maximum likelihood values are Omega_m*h = 0.264 +/-0.043, f_b = 0.286 +/- 0.065, sigma8 = 0.966 +/- 0.048, and beta = 0.45 +/- 0.12. When we take a prior on Omega_b from WMAP, we find Omega_m*h = 0.207 +/- 0.030, which is in excellent agreement with WMAP and 2dF. This indicates that we have reasonably measured the gross shape of the power spectrum but we have difficulty breaking the degeneracy between Omega_m*h and f_b because the baryon oscillations are not resolved in the current spectroscopic survey window function.

We measure cosmological parameters using the three-dimensional power spectrum P(k) from over 200,000 galaxies in the Sloan Digital Sky Survey (SDSS) in combination with WMAP and other data. Our results are consistent with a ``vanilla'' flat adiabatic Lambda-CDM model without tilt (n=1), running tilt, tensor modes or massive neutrinos. Adding SDSS information more than halves the WMAP-only error bars on some parameters, tightening 1 sigma constraints on the Hubble parameter from h~0.74+0.18-0.07 to h~0.70+0.04-0.03, on the matter density from Omega_m~0.25+/-0.10 to Omega_m~0.30+/-0.04 (1 sigma) and on neutrino masses from <11 eV to <0.6 eV (95%). SDSS helps even more when dropping prior assumptions about curvature, neutrinos, tensor modes and the equation of state. Our results are in substantial agreement with the joint analysis of WMAP and the 2dF Galaxy Redshift Survey, which is an impressive consistency check with independent redshift survey data and analysis techniques. In this paper, we place particular emphasis on clarifying the physical origin of the constraints, i.e., what we do and do not know when using different data sets and prior assumptions. For instance, dropping the assumption that space is perfectly flat, the WMAP-only constraint on the measured age of the Universe tightens from t0~16.3+2.3-1.8 Gyr to t0~14.1+1.0-0.9 Gyr by adding SDSS and SN Ia data. Including tensors, running tilt, neutrino mass and equation of state in the list of free parameters, many constraints are still quite weak, but future cosmological measurements from SDSS and other sources should allow these to be substantially tightened.

We measure the large-scale real-space power spectrum P(k) using a sample of 205,443 galaxies from the Sloan Digital Sky Survey, covering 2417 square degrees with mean redshift z~0.1. We employ a matrix-based method using pseudo-Karhunen-Loeve eigenmodes, producing uncorrelated minimum-variance measurements in 22 k-bands of both the clustering power and its anisotropy due to redshift-space distortions, with narrow and well-behaved window functions in the range 0.02 h/Mpc < k < 0.3h/Mpc. We pay particular attention to modeling, quantifying and correcting for potential systematic errors, nonlinear redshift distortions and the artificial red-tilt caused by luminosity-dependent bias. Our final result is a measurement of the real-space matter power spectrum P(k) up to an unknown overall multiplicative bias factor. Our calculations suggest that this bias factor is independent of scale to better than a few percent for k<0.1h/Mpc, thereby making our results useful for precision measurements of cosmological parameters in conjunction with data from other experiments such as the WMAP satellite. As a simple characterization of the data, our measurements are well fit by a flat scale-invariant adiabatic cosmological model with h Omega_m =0.201+/- 0.017 and L* galaxy sigma_8=0.89 +/- 0.02 when fixing the baryon fraction Omega_b/Omega_m=0.17 and the Hubble parameter h=0.72; cosmological interpretation is given in a companion paper.

We present a simple and accurate method to constrain galaxy bias based on the distribution of counts in cells. The most unique feature of our technique is that it is applicable to non-linear scales, where both dark matter statistics and the nature of galaxy bias are fairly complex. First, we estimate the underlying continuous distribution function from precise counts-in-cells measurements assuming local Poisson sampling. Then a robust, non-parametric inversion of the bias function is recovered from the comparison of the cumulative distributions in simulated dark matter and galaxy catalogs. Obtaining continuous statistics from the discrete counts is the most delicate novel part of our recipe. It corresponds to a deconvolution of a (Poisson) kernel. For this we present two alternatives: a model independent algorithm based on Richardson-Lucy iteration, and a solution using a parametric skewed lognormal model. We find that the latter is an excellent approximation for the dark matter distribution, but the model independent iterative procedure is more suitable for galaxies. Tests based on high resolution dark matter simulations and corresponding mock galaxy catalogs show that we can reconstruct the non-linear bias function down to highly non-linear scales with high precision in the range of $-1 \le \delta \le 5$. As far as the stochasticity of the bias, we have found a remarkably simple and accurate formula based on Poisson noise, which provides an excellent approximation for the scatter around the mean non-linear bias function. In addition we have found that redshift distortions have a negligible effect on our bias reconstruction, therefore our recipe can be safely applied to redshift surveys.

We present measurements of the angular cross-correlation between luminous red galaxies from the Sloan Digital Sky Survey and the cosmic microwave background temperature maps from the Wilkinson Microwave Anisotropy Probe. We find a statistically significant achromatic positive correlation between these two data sets, which is consistent with the expected signal from the late Integrated Sachs-Wolfe (ISW) effect. We do not detect any anti-correlation on small angular scales as would be produced from a large Sunyaev-Zel'dovich (SZ) effect, although we do see evidence for some SZ effect for our highest redshift samples. Assuming a flat universe, our preliminary detection of the ISW effect provides independent physical evidence for the existence of dark energy.

Understanding the clustering of galaxies has long been a goal of modern observational cosmology. Utilizing our photometric redshift technique a volume limited sample containing more than 2 million galaxies is constructed from the SDSS galaxy catalog. In the largest such analysis to date, we study the angular clustering as a function of luminosity and spectral type. Using Limber's equation we calculate the clustering length for the full data set as r0=5.77+/-0.10 Mpc/h. We find that r0 increases with luminosity by a factor of 1.6 over the sampled luminosity range, in agreement with previous redshift surveys. We also find that both the clustering length and the slope of the correlation function depend on the galaxy type. In particular, by splitting the galaxies in four groups by their rest-frame type we find a bimodal behavior in their clustering properties. Galaxies with spectral types similar to elliptical galaxies have a correlation length of 6.59 +/- 0.17 Mpc/h and a slope of the angular correlation function of 0.96 +/- 0.05 while blue galaxies have a clustering length of 4.51 +/- 0.19 Mpc/h and a slope of 0.68 +/- 0.09. The two intermediate color groups behave like their more extreme 'siblings', rather than showing a gradual transition in slope. We discuss these correlations in the context of current cosmological models for structure formation.

We introduce the theory of non-linear cosmological perturbations using the correspondence limit of the Schr\"odinger equation. The resulting formalism is equivalent to using the collisionless Boltzman (or Vlasov) equations which remain valid during the whole evolution, even after shell crossing. Other formulations of perturbation theory explicitly break down at shell crossing, e.g. Eulerean perturbation theory, which describes gravitational collapse in the fluid limit. This paper lays the groundwork by introducing the new formalism, calculating the perturbation theory kernels which form the basis of all subsequent calculations. We also establish the connection with conventional perturbation theories, by showing that third order tree level results, such as bispectrum, skewness, cumulant correlators, three-point function are exactly reproduced in the appropriate expansion of our results. We explicitly show that cumulants up to N=5 predicted by Eulerian perturbation theory for the dark matter field $\delta$ are exactly recovered in the corresponding limit. A logarithmic mapping of the field naturally arises in the Schr\"odinger context, which means that tree level perturbation theory translates into (possibly incomplete) loop corrections for the conventional perturbation theory. We show that the first loop correction for the variance is $\sigma^2 = \sigma_L^2+ (-1.14+n)\sigma_L^4$ for a field with spectral index $n$. This yields 1.86 and 0.86 for $n=-3,-2$ respectively, and to be compared with the exact loop order corrections 1.82, and 0.88. Thus our tree-level theory recovers the dominant part of first order loop corrections of the conventional theory, while including (partial) loop corrections to infinite order in terms of $\delta$.

The angular distribution of galaxies encodes a wealth of information about large scale structure. Ultimately, the Sloan Digital Sky Survey (SDSS) will record the angular positions of order 10^8 galaxies in five bands, adding significantly to the cosmological constraints. This is the first in a series of papers analyzing a rectangular stripe 2.5x90 degrees from early SDSS data. We present the angular correlation function for galaxies in four separate magnitude bins on angular scales ranging from 0.003 degrees to 15 degrees. Much of the focus of this paper is on potential systematic effects. We show that the final galaxy catalog -- with the mask accounting for regions of poor seeing, reddening, bright stars, etc. -- is free from external and internal systematic effects for galaxies brighter than r* = 22. Our estimator of the angular correlation function includes the effects of the integral constraint and the mask. The full covariance matrix of errors in these estimates is derived using mock catalogs with further estimates using a number of other methods.

We compute the angular power spectrum C_l from 1.5 million galaxies in early SDSS data on large angular scales, l<600. The data set covers about 160 square degrees, with a characteristic depth of order 1 Gpc/h in the faintest (21<r<22) of our four magnitude bins. Cosmological interpretations of these results are presented in a companion paper by Dodelson et al (2001). The data in all four magnitude bins are consistent with a simple flat ``concordance'' model with nonlinear evolution and linear bias factors of order unity. Nonlinear evolution is particularly evident for the brightest galaxies. A series of tests suggest that systematic errors related to seeing, reddening, etc., are negligible, which bodes well for the sixtyfold larger sample that the SDSS is currently collecting. Uncorrelated error bars and well-behaved window functions make our measurements a convenient starting point for cosmological model fitting.

We present initial results for counts in cells statistics of the angular distribution of galaxies in early data from the Sloan Digital Sky Survey (SDSS). We analyze a rectangular stripe $2.5^\circ$ wide, covering approximately 160 sq. degrees, containing over $10^6$ galaxies in the apparent magnitude range $18 < r^\prime < 22$, with areas of bad seeing, contamination from bright stars, ghosts, and high galactic extinction masked out. This survey region, which forms part of the SDSS Early Data Release, is the same as that for which two-point angular clustering statistics have recently been computed. The third and fourth moments of the cell counts, $s_3$ (skewness) and $s_4$ (kurtosis), constitute the most accurate measurements to date of these quantities (for $r^\prime < 21$) over angular scales $0.015^\circ-0.3^\circ$. They display the approximate hierarchical scaling expected from non-linear structure formation models and are in reasonable agreement with the predictions of $\Lambda$-dominated cold dark matter models with galaxy biasing that suppresses higher order correlations at small scales. The results are in general consistent with previous measurements in the APM, EDSGC, and Deeprange surveys. These results suggest that the SDSS imaging data are free of systematics to a high degree and will therefore enable determination of the skewness and kurtosis to 1% and less then 10%, as predicted by Colombi, Szapudi, & Szalay (1998).

I present here a review of past and present multi-disciplinary research of the Pittsburgh Computational AstroStatistics (PiCA) group. This group is dedicated to developing fast and efficient statistical algorithms for analysing huge astronomical data sources. I begin with a short review of multi-resolutional kd-trees which are the building blocks for many of our algorithms. For example, quick range queries and fast n-point correlation functions. I will present new results from the use of Mixture Models (Connolly et al. 2000) in density estimation of multi-color data from the Sloan Digital Sky Survey (SDSS). Specifically, the selection of quasars and the automated identification of X-ray sources. I will also present a brief overview of the False Discovery Rate (FDR) procedure (Miller et al. 2001a) and show how it has been used in the detection of ``Baryon Wiggles'' in the local galaxy power spectrum and source identification in radio data. Finally, I will look forward to new research on an automated Bayes Network anomaly detector and the possible use of the Locally Linear Embedding algorithm (LLE; Roweis & Saul 2000) for spectral classification of SDSS spectra.

The DEEP2 project will obtain redshifts for ~60,000 galaxies between z~0.7-1.5 in a comoving volume of 7 10^6 Mpc/h^3 for an LCDM universe. The survey will map four separate 2 by 0.5 degree strips of the sky. To study the expected clustering, we have constructed mock galaxy catalogs from the GIF Virgo Consortium simulations. We present two- and three-point correlation analyses of these mock galaxy catalogs to test how well we will measure these statistics in the presence of selection biases which will limit the surface density of galaxies which we can select for spectroscopy. We find that neither the two-point nor three-point correlation functions are significantly compromised. We will be able to make simple corrections to account for the small amount of bias introduced. We quantify the expected redshift distortions due to random orbital velocities of galaxies within groups and clusters on small scales of ~1 Mpc/h using the pairwise velocity dispersion sigma_12 and galaxy-weighted velocity dispersion sigma_1, which we are able to measure within ~10%. We also estimate the expected large-scale coherent infall of galaxies due to supercluster formation (``Kaiser effect''). From this measure we will be able to constrain beta to within ~0.1 at z=1. For the DEEP2 survey we will combine the correlation statistics with galaxy observables such as spectral type, morphology, absolute luminosity, and linewidth to measure the relative biases in different galaxy types. Here we use a counts-in-cells analysis to measure sigma_8 as a function of redshift and determine the relative bias between galaxy samples based on absolute luminosity. We expect to measure sigma_8 to within 10% and detect the evolution of relative bias with redshift at the 4-5 sigma level.

We present measurements of parameters of the 3-dimensional power spectrum of galaxy clustering from 222 square degrees of early imaging data in the Sloan Digital Sky Survey. The projected galaxy distribution on the sky is expanded over a set of Karhunen-Loeve eigenfunctions, which optimize the signal-to-noise ratio in our analysis. A maximum likelihood analysis is used to estimate parameters that set the shape and amplitude of the 3-dimensional power spectrum. Our best estimates are Gamma=0.188 +/- 0.04 and sigma_8L = 0.915 +/- 0.06 (statistical errors only), for a flat Universe with a cosmological constant. We demonstrate that our measurements contain signal from scales at or beyond the peak of the 3D power spectrum. We discuss how the results scale with systematic uncertainties, like the radial selection function. We find that the central values satisfy the analytically estimated scaling relation. We have also explored the effects of evolutionary corrections, various truncations of the KL basis, seeing, sample size and limiting magnitude. We find that the impact of most of these uncertainties stay within the 2-sigma uncertainties of our fiducial result.

The Sloan Digital Sky Survey is one of the first multicolor photometric and spectroscopic surveys designed to measure the statistical properties of galaxies within the local Universe. In this Letter we present some of the initial results on the angular 2-point correlation function measured from the early SDSS galaxy data. The form of the correlation function, over the magnitude interval 18<r*<22, is shown to be consistent with results from existing wide-field, photographic-based surveys and narrower CCD galaxy surveys. On scales between 1 arcminute and 1 degree the correlation function is well described by a power-law with an exponent of ~ -0.7. The amplitude of the correlation function, within this angular interval, decreases with fainter magnitudes in good agreement with analyses from existing galaxy surveys. There is a characteristic break in the correlation function on scales of approximately 1-2 degrees. On small scales, < 1', the SDSS correlation function does not appear to be consistent with the power-law form fitted to the 1'< theta <0.5 deg data. With a data set that is less than 2% of the full SDSS survey area, we have obtained high precision measurements of the power-law angular correlation function on angular scales 1' < theta < 1 deg, which are robust to systematic uncertainties. Because of the limited area and the highly correlated nature of the error covariance matrix, these initial results do not yet provide a definitive characterization of departures from the power-law form at smaller and larger angles. In the near future, however, the area of the SDSS imaging survey will be sufficient to allow detailed analysis of the small and large scale regimes, measurements of higher-order correlations, and studies of angular clustering as a function of redshift and galaxy type.

Early photometric data from the Sloan Digital Sky Survey (SDSS) contain angular positions for 1.5 million galaxies. In companion papers, the angular correlation function $w(\theta)$ and 2D power spectrum $C_l$ of these galaxies are presented. Here we invert Limber's equation to extract the 3D power spectrum from the angular results. We accomplish this using an estimate of $dn/dz$, the redshift distribution of galaxies in four different magnitude slices in the SDSS photometric catalog. The resulting 3D power spectrum estimates from $w(\theta)$ and $C_l$ agree with each other and with previous estimates over a range in wavenumbers $0.03 < k/{\rm h Mpc}^{-1} < 1$. The galaxies in the faintest magnitude bin ($21 < \rstar < 22$, which have median redshift $z_m=0.43$) are less clustered than the galaxies in the brightest magnitude bin ($18 < \rstar < 19$ with $z_m=0.17$), especially on scales where nonlinearities are important. The derived power spectrum agrees with that of Szalay et al. (2001) who go directly from the raw data to a parametric estimate of the power spectrum. The strongest constraints on the shape parameter $\Gamma$ come from the faintest galaxies (in the magnitude bin $21 < \rstar < 22$), from which we infer $\Gamma = 0.14^{+0.11}_{-0.06}$ (95% C.L.).

Szapudi et al (2001) introduced the method of estimating angular power spectrum of the CMB sky via heuristically weighted correlation functions. Part of the new technique is that all (co)variances are evaluated by massive Monte Carlo simulations, therefore a fast way to measure correlation functions in a high resolution map is essential. This letter presents a new algorithm to calculate pixel space correlation functions via fast spherical harmonics transforms. Our present implementation of the idea extracts correlations from a MAP-like CMB map (HEALPix resolution of 512, i.e. $ \simeq 3 \times 10^6$ pixels) in about 5 minutes on a 500MHz computer, including $C_\ell$ inversion; the analysis of one Planck-like map takes less then one hour. We use heuristic window and noise weighting in pixel space, and include the possibility of additional signal weighting as well, either in $\ell$ or pixel space. We apply the new code to an ensemble of MAP simulations, to test the response of our method to the inhomogenous sky coverage/noise of MAP. We show that the resulting $C_\ell$'s are very close to the theoretical expectations. The HEALPix based implementation of the method, SpICE (Spatially Inhomogenous Correlation Estimator) will be available to the public from the authors.

We present here a new algorithm for the fast computation of N-point correlation functions in large astronomical data sets. The algorithm is based on kdtrees which are decorated with cached sufficient statistics thus allowing for orders of magnitude speed-ups over the naive non-tree-based implementation of correlation functions. We further discuss the use of controlled approximations within the computation which allows for further acceleration. In summary, our algorithm now makes it possible to compute exact, all-pairs, measurements of the 2, 3 and 4-point correlation functions for cosmological data sets like the Sloan Digital Sky Survey (SDSS; York et al. 2000) and the next generation of Cosmic Microwave Background experiments (see Szapudi et al. 2000).

We outline here the next generation of cluster-finding algorithms. We show how advances in Computer Science and Statistics have helped develop robust, fast algorithms for finding clusters of galaxies in large multi-dimensional astronomical databases like the Sloan Digital Sky Survey (SDSS). Specifically, this paper presents four new advances: (1) A new semi-parametric algorithm - nicknamed ``C4'' - for jointly finding clusters of galaxies in the SDSS and ROSAT All-Sky Survey databases; (2) The introduction of the False Discovery Rate into Astronomy; (3) The role of kernel shape in optimizing cluster detection; (4) A new determination of the X-ray Cluster Luminosity Function which has bearing on the existence of a ``deficit'' of high redshift, high luminosity clusters. This research is part of our ``Computational AstroStatistics'' collaboration (see Nichol et al. 2000) and the algorithms and techniques discussed herein will form part of the ``Virtual Observatory'' analysis toolkit.

We present a counts-in-cells analysis of clustering in the optically selected Durham/UKST and Stromlo-APM Galaxy Redshift Surveys. Minimum variance estimates of the second moment, skewness (S_3) and kurtosis (S_4) of the count probability distribution are extracted from a series of volume limited samples of varying radial depth. The corresponding theoretical error calculation takes into account all sources of statistical error on the measurement of the moments, and is in good agreement with the dispersion over mock redshift catalogues. The errors that we find on S_3 and S_4 are larger than those quoted in previous studies, in spite of the fact that the surveys we consider cover larger volumes. S_3 varies little with cell size, with values in the range 1.8-2.2 and errors ~20%, for cubical cells of side 3-20 h^{-1}Mpc. Direct measurements of S_3 are possible out to ~35 h^{-1}Mpc, though with larger errors. A significant determination of S_4 is only possible for one scale, l=6 h^{-1}Mpc, with S_4~5. We compare our results with theoretical predictions from N-body simulations of cold dark matter universes. Qualitatively, the skewness of the dark matter has the same form as that of the galaxies. However, the amplitude of the galaxy S_3 is lower than that predicted for the dark matter. Our measurements of S_3 are consistent with the predictions of a simple model in which initially Gaussian fluctuations in the dark matter evolve gravitationally, if a second order bias term is specified, in addition to the traditional linear bias, in order to describe the relation between the distribution of galaxies and dark matter.

We propose and implement a fast, universally applicable method for extracting the angular power spectrum C_l from CMB temperature maps by first estimating the correlation function \xi(\theta). Our procedure recovers the C_l's using N^2 (but potentially N logN), operations, where N is the number of pixels. This is in contrast with standard maximum likelihood techniques which require N^3 operations. Our method makes no special assumptions about the map, unlike present fast techniques which rely on symmetries of the underlying noise matrix, sky coverage, scanning strategy, and geometry. This enables for the first time the analysis of megapixel maps without symmetries. The key element of our technique is the accurate multipole decomposition of \xi(\theta). The C_l error bars and cross-correlations are found by a Monte-Carlo approach. We applied our technique to a large number of simulated maps with Boomerang sky coverage in 81000 pixels. We used a diagonal noise matrix, with approximately the same amplitude as Boomerang. These studies demonstrate that our technique provides an unbiased estimator of the C_l's. Even though our method is approximate, the error bars obtained are nearly optimal, and converged only after few tens of Monte-Carlo realizations. Our method is directly applicable for the non-diagonal noise matrix. This, and other generalizations, such as minimum variance weighting schemes, polarization, and higher order statistics are also discussed.

Constraints on the validity of the hierarchical gravitational instability theory and the evolution of biasing are presented based upon measurements of higher order clustering statistics in the Deeprange Survey, a catalog of $\sim710,000$ galaxies with $I_{AB} \le 24$ derived from a KPNO 4m CCD imaging survey of a contiguous $4^{\circ} \times 4^{\circ}$ region. We compute the 3-point and 4-point angular correlation functions using a direct estimation for the former and the counts-in-cells technique for both. The skewness $s_3$ decreases by a factor of $\simeq 3-4$ as galaxy magnitude increases over the range $17 \le I \le 22.5$ ($0.1 \lesssim z \lesssim 0.8$). This decrease is consistent with a small {\it increase} of the bias with increasing redshift, but not by more than a factor of 2 for the highest redshifts probed. Our results are strongly inconsistent, at about the $3.5-4 \sigma$ level, with typical cosmic string models in which the initial perturbations follow a non-Gaussian distribution - such models generally predict an opposite trend in the degree of bias as a function of redshift. We also find that the scaling relation between the 3-point and 4-point correlation functions remains approximately invariant over the above magnitude range. The simplest model that is consistent with these constraints is a universe in which an initially Gaussian perturbation spectrum evolves under the influence of gravity combined with a low level of bias between the matter and the galaxies that decreases slightly from $z \sim 0.8$ to the current epoch.

The fully general calculation of the cosmic error on N-point correlation functions and related quantities is presented. More precisely, the variance caused by the finite volume, discreteness, and edge effects is determined for {\em any} estimator which is based on a general function of N-tuples, such as multi-point correlation functions and multi-spectra. The results are printed explicitly for the two-point correlation function (or power-spectrum), and for the three-point correlation (or bispectrum). These are the most popular statistics in the study of large scale structure, yet, the a general calculation of their variance has not been performed until now.

We investigate the spatial clustering of galaxies in the PSCz galaxy redshift survey, as revealed by the two-point correlation function, the luminosity mark correlations, and the moments of counts-in-cells. We construct volume-limited subsamples at different depths, and search for a luminosity dependence of the clustering pattern. We find no statistically significant effect in either the two-point correlation function or the mark correlations and so we take each subsample (of different characteristic luminosity) as representing the same statistical process. We then carry out a counts-in-cells analysis of the volume-limited subsamples, including a rigorous error calculation based on the recent theory of Szapudi, Colombi and Bernardeau. In this way, we derive the best estimates to date of the skewness and kurtosis of IRAS galaxies in redshift space. Our results agree well with previous measurements in both the parent angular catalogue, and in the derived redshift surveys. This is in contrast with smaller, optically selected surveys, were there is a discrepancy between the redshift space and projected measurements. Predictions from cold dark matter theory, obtained using the recent semi-analytical model of galaxy formation of Benson {\it et al}, provide an excellent description of our clustering data.

Nine of the most important estimators known for the two-point correlation function are compared using a predetermined, rigorous criterion. The indicators were extracted from over 500 subsamples of the Virgo Hubble Volume simulation cluster catalog. The ``real'' correlation function was determined from the full survey in a 3000Mpc/h periodic cube. The estimators were ranked by the cumulative probability of returning a value within a certain tolerance of the real correlation function. This criterion takes into account bias and variance, and it is independent of the possibly non-Gaussian nature of the error statistics. As a result for astrophysical applications a clear recommendation has emerged: the Landy & Szalay (1993) estimator, in its original or grid version Szapudi & Szalay (1998), are preferred in comparison to the other indicators examined, with a performance almost indistinguishable from the Hamilton (1993) estimator.

The errors on statistics measured in finite galaxy catalogs are exhaustively investigated. The theory of errors on factorial moments by Szapudi & Colombi (1996) is applied to cumulants via a series expansion method. All results are subsequently extended to the weakly non-linear regime. Together with previous investigations this yields an analytic theory of the errors for moments and connected moments of counts in cells from highly nonlinear to weakly nonlinear scales. The final analytic formulae representing the full theory are explicit but somewhat complicated. Therefore as a companion to this paper we supply a FORTRAN program capable of calculating the described quantities numerically (abridged).

Counts-in-cells are measured in the $\tau$CDM Virgo Hubble Volume simulation. This large N-body experiment has 10^9 particles in a cubic box of size 2000 h^{-1} Mpc. The unprecedented combination of size and resolution allows for the first time a realistic numerical analysis of the cosmic errors and cosmic correlations of statistics related to counts-in-cells measurements, such as the probability distribution function P_N itself, its factorial moments F_k and the related cumulants $\xiav$ and S_N's. These statistics are extracted from the whole simulation cube, as well as from 4096 sub-cubes of size 125 h^{-1}Mpc, each representing a virtual random realization of the local universe. The measurements and their scatter over the sub-volumes are compared to the theoretical predictions of Colombi, Bouchet & Schaeffer (1995) for P_0, and of Szapudi & Colombi (1996, SC) and Szapudi, Colombi & Bernardeau (1999a, SCB) for the factorial moments and the cumulants. The general behavior of experimental variance and cross-correlations as functions of scale and order is well described by theoretical predictions, with a few percent accuracy in the weakly non-linear regime for the cosmic error on factorial moments. (... more in paper >...)

Colombi et al. 1999 (paper I) investigated the counts-in-cells statistics and their respective errors in the $\tau$CDM Virgo Hubble Volume simulation. This extremely large $N$-body experiment also allows a numerical investigation of the {\em cosmic distribution function}, $\Upsilon(\tA)$ itself for the first time. For a statistic $A$, $\Upsilon(\tA)$ is the probability density of measuring the value $\tA$ in a finite galaxy catalog. $\Upsilon$ was evaluated for the distribution of counts-in-cells, $P_N$, the factorial moments, $F_k$, and the cumulants, $\xiav$ and $S_N$'s, using the same subsamples as paper I. While paper I concentrated on the first two moments of $\Upsilon$, i.e. the mean, the cosmic error and the cross-correlations, here the function $\Upsilon$ is studied in its full generality, including a preliminary analysis of joint distributions $\Upsilon(\tA,\tB)$. The most significant, and reassuring result for the analyses of future galaxy data is that the cosmic distribution function is nearly Gaussian provided its variance is small. A good practical criterion for the relative cosmic error is that $\Delta A/A \la 0.2$. This means that for accurate measurements, the theory of the cosmic errors, presented by Szapudi & Colombi (1996) and Szapudi, Colombi & Bernardeau (1999), and confirmed empirically by paper I, is sufficient for a full statistical description and thus for a maximum likelihood rating of models. As the cosmic error increases, the cosmic distribution function $\Upsilon$ becomes increasingly skewed and is well described by a generalization of the lognormal distribution. The cosmic skewness is introduced as an additional free parameter. (...more in paper...)

We describe a set of new estimators for the N-point correlation functions of point processes. The variance of these estimators is calculated for the Poisson and binomial cases. It is shown that the variance of the unbiased estimator converges to the continuum value much faster than with any previously used alternative, all terms with slower convergence exactly cancel. We compare our estimators with Ripley's $\tK_0$ and $\tK_2$.

This work presents a set of new statistics, the cumulant correlators, aimed at high precision analysis of the galaxy distribution. They form a symmetric matrix, $Q_{NM}$, related to moment correlators the same way as cumulants are related to the moments of the distribution. They encode more information than the usual cumulants, $S_N$'s, and their extraction from data is similar to the calculation of the two-point correlation function. Perturbation theory (PT), its generalization, the extended perturbation theory (EPT), and the hierarchical assumption (HA) have simple predictions for these statistics. As an example, the factorial moment correlators measured by Szapudi, Dalton, Efstathiou & Szalay (1996) in the APM catalog are reanalyzed using this technique. While the previous analysis assumed hierarchical structure constants, this method can directly investigate the validity of HA, along with PT, and EPT. The results in agreement with previous findings indicate that, at the small scales used for this analysis, the APM data supports HA. When all non-linear corrections are taken into account it is a good approximation at the 20 percent level. It appears that PT, and a natural generalization of EPT for cumulant correlators does not provide such a good fit for the APM at small scales. Once the validity the HA is approximately established, cumulant correlators can separate the amplitudes of different tree-types in the hierarchy up to fifth order. As an example, the weights for the fourth order tree topologies are calculated including all non-linear corrections.

Higher order statistics are investigated in ($\Omega$)CDM universes by analyzing $500\mpc$ high resolution tree N-body simulations with both $\Omega = 1$, and $\Omega < 1$. The amplitudes of the N-point correlation functions are calculated from moments of counts-in-cells determined by a pair of new algorithms especially developed for large simulations. This approach enables massive oversampling with $\simeq 10^{9-14}$ cells for accurate determination of factorial moments from up to 47 million particles in the scale range of $8 \kpc - 125\mpc$. Thorough investigation shows that there are three scale ranges in the simulations: $\ge 8\mpc$, weakly non-linear regime, where perturbation theory applies with utmost precision, $1\mpc - 8\mpc$, the non-linear plateau, and finally $\le 1\mpc$, a regime where dynamical discreteness effects dominate the higher order statistics. In the physically relevant range of $1\mpc-125\mpc$ the results i) confirm the validity of perturbation theory in the weakly non-linear regime, ii) establish the existence of a plateau in the highly non-linear regime similar to the one observed in scale free simulations iii) show extended perturbation theory to be an excellent approximation for the non-linear regime iv) find the time dependence of the $S_N$'s to be negligible in both regimes v) in comparison with similar measurements in the EDSGC survey, strongly support $\Omega < 1$ with no biasing vi) show that the formulae of Szapudi & Colombi (1996) provide a good approximation for errors on higher order statistics measured in N-body simulations.

For a given statistic, A, the cosmic distribution function, Upsilon(VA), is the probability of measuring a value VA in a finite galaxy catalog. For statistics related to count-in-cells, such as factorial moments, F_k, the average correlation function, xiav, and cumulants, S_N, the functions Upsilon(VF_k), Upsilon(Vxiav), and Upsilon(VS_N) were measured in a large tauCDM simulation. This N-body experiment simulates almost the full ``Hubble Volume'' of the universe, thus, for the first time, it allowed for an accurate analysis of the cosmic distribution function, and, in particular, of its variance (Delta A)^2, the cosmic error. The resulting detailed knowledge about the shape of Upsilon is crucial for likelihood analyses. The measured cosmic error agrees remarkably well with the theoretical predictions of Szapudi & Colombi (1996) and Szapudi, Bernardeau & Colombi (1998) in the weakly non-linear regime, while the predictions are slightly above the measurements in the highly nonlinear regime. When the relative cosmic error is small, (Delta A/A)^2, function Upsilon is nearly Gaussian. When (Delta A/A)^2 approaches unity or is larger, function Upsilon(VA) is increasingly skewed and well approximated by a lognormal distribution for A=F_k, or A=xiav. The measured cumulants follow accurately the perturbation theory predictions in the weakly nonlinear regime. Extended perturbation theory is an excellent approximation for all the available dynamic range.

We present constraints on the evolution of large-scale structure from a catalog of 710,000 galaxies with I_AB <= 24 derived from a KPNO 4m CCD imaging survey of a contiguous 4 deg x 4 deg region. The advantage of using large contiguous surveys for measuring clustering properties on even modest angular scales is substantial: the effects of cosmic scatter are strongly suppressed. We provide highly accurate measurements of the two-point angular correlation function, w(theta), as a function of magnitude on scales up to 1.5 degrees. The amplitude of w(theta) declines by a factor of ~10 over the range 16 <= I <= 20 but only by a factor of 2 - 3 over the range 20 < I <= 23. For a redshift dependence of the spatial correlation function, xi(r), parameterized as xi(r,z)=(r/r_o)^(-gamma)(1 + z)^(-[3+epsilon]), we find r_o=5.2 +/- 0.4 Mpc/h, and epsilon >= 0 for I <= 20. This is in good agreement with the results from local redshift surveys. At I > 20, our best fit values shift towards lower r_o and more negative epsilon. A strong covariance between r_o and epsilon prevent us from rejecting epsilon > 0 even at faint magnitudes but if epsilon > 1, we strongly reject r_o <= 4/h Mpc (co-moving). The above expression for xi(r,z) and our data give a correlation length of r_o(z=0.5) approx 3.0 +/- 0.4 Mpc/h, about a factor of 2 larger than the correlation length at z = 0.5 derived from the Canada--France Redshift Survey (CFRS). The small volume sampled by the CFRS and other deep redshift probes, however, make these spatial surveys strongly susceptible to cosmic scatter and will tend to bias their derived correlation lengths low. Our galaxy counts agree well with those from the HDF survey and, thus, argue against a significant inclusion of sub-galactic components in the latter census for I < 24.

The first non-trivial cumulant correlator of the galaxy density field $Q_{21}$ is examined from the point of view of biasing. It is shown that to leading order it depends on two biasing parameters $b$, and $b_2$, and on $q_{21}$, the underlying cumulant correlator of the mass. As the skewness $Q_3$ has analogous properties, the slope of the correlation function $-\gamma$, $Q_3$, and $Q_{21}$ uniquely determine the bias parameter on a particular scale to be $b = \gamma/6(Q_{21}-Q_3)$, when working in the context of gravitational instability with Gaussian initial conditions. Thus on large scales, easily accessible with the future Sloan Digital Sky Survey and the 2 Degree Field Survey, it will be possible to extract $b$, and $b_2$ from simple counts in cells measurements. Moreover, the higher order cumulants, $Q_N$, successively determine the higher order biasing parameters. From these it is possible to predict higher order cumulant correlators as well. Comparing the predictions with the measurements will provide internal consistency checks on the validity of the assumptions in the theory, most notably perturbation theory of the growth of fluctuations by gravity and Gaussian initial conditions. Since the method is insensitive to $\Omega$, it can be successfully combined with results from velocity fields, which determine $\Omega^{0.6}/b$, to measure the total density parameter in the universe.

Using an Eulerian perturbative calculation, we show that the distribution of relative pairwise velocities which arises from gravitational instability of Gaussian density fluctuations has asymmetric (skewed) exponential tails. The negative skewness is induced by the negative mean streaming velocity of pairs (the infall prevails over expansion), while the exponential tails arise because the relative pairwise velocity is a number, not volume weighted statistic. The derived probability distribution is compared with N-body simulations and shown to provide a reasonable fit.

Clustering statistics are compared in the Automatic Plate Machine (APM) and the Edinburgh/Durham Southern Galaxy Catalogue (EDSGC) angular galaxy surveys. Both surveys were independently constructed from scans of the same adjacent UK IIIa--J Schmidt photographic plates with the APM and COSMOS microdensitometers, respectively. The comparison of these catalogs is a rare practical opportunity to study systematic errors, which cannot be achieved via simulations or theoretical methods. On intermediate scales, $0.1^\circ < \theta < 0.5^\circ$, we find good agreement for the cumulants or reduced moments of counts in cells up to sixth order. On larger scales there is a small disagreement due to edge effects in the EDSGC, which covers a smaller area. On smaller scales, we find a significant disagreement that can only be attributed to differences in the construction of the surveys, most likely the dissimilar deblending of crowded fields. The overall agreement of the APM and EDSGC is encouraging, and shows that the results for intermediate scales should be fairly robust. On the other hand, the systematic deviations found at small scales are significant in a regime, where comparison with theory and simulations is possible. This is an important fact to bear in mind when planning the construction of future digitized galaxy catalogs.

In the near future a new generation of CCD based galaxy surveys will enable high precision determination of the N-point correlation functions. The resulting information will help to resolve the ambiguities associated with two-point correlation functions thus constraining theories of structure formation, biasing, and Gaussianity of initial conditions independently of the value of $\Omega$. As one the most successful methods to extract the amplitude of higher order correlations is based on measuring the distribution of counts in cells, this work presents an advanced way of measuring it with unprecedented accuracy. Szapudi and Colombi (1996, hereafter \cite{sc96}) identified the main sources of theoretical errors in extracting counts in cells from galaxy catalogs. One of these sources, termed as measurement error, stems from the fact that conventional methods use a finite number of sampling cells to estimate counts in cells. This effect can be circumvented by using an infinite number of cells. This paper presents an algorithm, which, in practice achieves this goal, i.e. it is equivalent to throwing an infinite number of sampling cells in finite time. The errors associated with sampling cells are completely eliminated by this procedure which will be essential for the accurate analysis of future surveys.

The effects of sampling are investigated on measurements of counts-in-cells in three-dimensional magnitude limited galaxy surveys, with emphasis on moments of the underlying smooth galaxy density field convolved with a spherical window. A new estimator is proposed for measuring the k-th order moment < rho^k >: the weighted factorial moment F_k[w], corrected for the effects of the varying selection function. The cosmic error on the measurement of F_k[w] is computed via the the formalism of Szapudi & Colombi (1996), which is generalized to include selection effects. The integral equation for finding the minimum variance weight is solved numerically, and an intuitive analytical approximation is derived. The resulting estimator is more accurate than the traditional method of counts-in-cells in volume limited samples, which discards useful information. As a practical example we consider the case of the future Sloan Digital Sky Survey. Optimal (sparse) sampling strategies for designing magnitude limited redshift surveys are investigated as well. It is found that the optimal strategy depends greatly on the statistics and scales considered. Finally we consider the issue of designing the geometry of a catalog, when it covers only a small fraction of the sky.

We have evaluated a systematic effect on counts-in-cells analysis of deep, wide-field galaxy catalogues induced by the evolution of clustering within the survey volume. A multiplicative correction factor is explicitly presented, which can be applied after the higher order correlation functions have been extracted in the usual way, without taking into account the evolution. The general theory of this effect combined with the ansatz describing the non-linear evolution of clustering in simulations enables us to estimate the magnitude of the correction factor in different cosmologies. In a series of numerical calculations assuming an array of cold dark matter models, it is found that, as long as galaxies are unbiased tracers of underlying density field, the effect is relatively small ($ \simeq 10%$) for the shallow surveys ($z <0.2$), while it becomes significant (order of unity) in deep surveys ($z \sim1$). Depending on the scales of interest, the required correction is comparable to or smaller than the expected errors of on-going wide-field galaxy surveys such as the SDSS and 2dF. Therefore at present, the effect has to be taken into account for high precision measurements at very small scales only, while in future deep surveys it amounts to a significant correction.

A class of improved estimators is proposed for N-point correlation functions of galaxy clustering, and for discrete spatial random processes in general. In the limit of weak clustering, the variance of the unbiased estimator converges to the continuum value much faster than with any alternative, all terms giving rise to a slower convergence exactly cancel. Explicit variance formulae are provided for both Poisson and multinomial point processes using techniques for spatial statistics reported by Ripley (1988). The formalism naturally includes most previously used statistical tools such as N-point correlation functions and their Fourier counterparts, moments of counts-in-cells, and moment correlators. For all these, and perhaps some other statistics our estimator provides a straightforward means for efficient edge corrections.

Counts in cells are used to analyse the higher order properties of the statistics of the EDSGC survey. The probability distribution is obtained from an equal area projection source catalog with infinite oversampling over the range $0.015^\circ-2^\circ$. The factorial moments of the resulting distribution and the $s_N$'s characterizing the non-Gaussian nature of the distribution are extracted. These results are compared to previous results from the APM survey, and to theoretical results from perturbation theory. The deprojected 3D values corresponding to the $s_N$'s are also determined. We find that the 3D values match the scaling relation for strongly nonlinear clustering found in N-body simulations remarkably well, suggesting that the galaxies are reliable tracers of the underlying mass distribution.

We examine the errors on counts in cells extracted from galaxy surveys. The measurement error, related to the finite number of sampling cells, is disentangled from the ``cosmic error'', due to the finiteness of the survey. Using the hierarchical model and assuming locally Poisson behavior, we identified three contributions to the cosmic error: The finite volume effect is proportional to the average of the two-point correlation function over the whole survey. It accounts for possible fluctuations of the density field at scales larger than the sample size. The edge effect is related to the geometry of the survey. It accounts for the fact that objects near the boundary carry less statistical weight than those further away from it. The discreteness effect is due to the fact that the underlying smooth random field is sampled with finite number of objects. This is the ``shot noise'' error. Measurements of errors in artificial hierarchical samples showed excellent agreement with our predictions. The probability distribution of errors is increasingly skewed when the order $N$ and/or the cell size increases. The Gaussian approximation is valid only in the weakly non-linear regime, otherwise it severely underestimates the true errors. We study the concept of ``number of statistically independent cells'' This number is found to depend highly on the statistical object under study and is generally quite different from the number of cells needed to cover the survey volume. In light of these findings, we advocate high oversampling for measurements of counts in cells.

We investigate in detail the effects of sampling on our ability to accurately reconstruct the distribution of galaxies from galaxy surveys. We use a simple probability theory approach, Bayesian classifier theory and Bayesian transition probabilities. We find the best Bayesian estimator for the case of low sampling rates, and show that even in the optimal case certain higher order characteristics of the distribution are irretrievably washed out by sparse sampling: we illustrate this by a simple model for cluster selection. We show that even choosing an optimal threshold, there are nonzero numbers for both misidentified clusters and true clusters missed. The introduction of sampling has an effect on the distribution function that is similar to convolution. Deconvolution is possible and given in the paper, although it might become unstable as sampling rates become low. These findings have important consequences on planning and strategies of future galaxy surveys.

We apply a new statistics, the factorial moment correlators, to density maps obtained from the APM survey. The resulting correlators are all proportional to the two point correlation function, substantially amplified, with an amplification nearly exponential with the total rank of the correlators. This confirms the validity of the hierarchical clustering assumption on the dynamic range examined, corresponding to $0.5 \hmpc - 50 \hmpc$ in three dimensional space. The Kirkwood superposition with loop terms is strongly rejected. The structure coefficients of the hierarchy are also fitted. The high quality of the APM catalog enabled us to disentangle the various contributions from the power spectrum, small scale nonlinear clustering, and combinatorial effects, all of which affect the amplification of the correlators. These effects should appear in correlations of clusters in a similar fashion.