We investigate the range of applicability of a model for the real-space power spectrum based on N-body dynamics and a (quadratic) Lagrangian bias expansion. This combination uses the highly accurate particle displacements that can be efficiently achieved by modern N-body methods with a symmetries-based bias expansion which describes the clustering of any tracer on large scales. We show that at low redshifts, and for moderately biased tracers, the substitution of N-body-determined dynamics improves over an equivalent model using perturbation theory by more than a factor of two in scale, while at high redshifts and for highly biased tracers the gains are more modest. This hybrid approach lends itself well to emulation. By removing the need to identify halos and subhalos, and by not requiring any galaxy-formation-related parameters to be included, the emulation task is significantly simplified at the cost of modeling a more limited range in scale.

This paper introduces the Hidden Valley simulations, a set of trillion-particle N-body simulations in gigaparsec volumes aimed at intensity mapping science. We present details of the simulations and their convergence, then specialize to the study of 21-cm fluctuations between redshifts 2 and 6. Neutral hydrogen is assigned to halos using three prescriptions, and we investigate the clustering in real and redshift-space at the 2-point level. In common with earlier work we find the bias of HI increases from near 2 at z = 2 to 4 at z = 6, becoming more scale dependent at high z. The level of scale-dependence and decorrelation with the matter field are as predicted by perturbation theory. Due to the low mass of the hosting halos, the impact of fingers of god is small on the range relevant for proposed 21-cm instruments. We show that baryon acoustic oscillations and redshift-space distortions could be well measured by such instruments. Taking advantage of the large simulation volume, we assess the impact of fluctuations in the ultraviolet background, which change HI clustering primarily at large scales.

We study the impact of lensing corrections on modeling cross correlations between CMB lensing and galaxies, cosmic shear and galaxies, and galaxies in different redshift bins. Estimating the importance of these corrections becomes necessary in the light of anticipated high-accuracy measurements of these observables. While higher order lensing corrections (sometimes also referred to as post Born corrections) have been shown to be negligibly small for lensing auto correlations, they have not been studied for cross correlations. We evaluate the contributing four-point functions without making use of the Limber approximation and compute line-of-sight integrals with the numerically stable and fast FFTlog formalism. We find that the relative size of lensing corrections depends on the respective redshift distributions of the lensing sources and galaxies, but that they are generally small for high signal-to-noise correlations. We point out that a full assessment and judgement of the importance of these corrections requires the inclusion of lensing Jacobian terms on the galaxy side. We identify these additional correction terms, but do not evaluate them due to their large number. We argue that they could be potentially important and suggest that their size should be measured in the future with ray-traced simulations. We make our code publicly available.

Upcoming 21-cm intensity surveys will use the hyperfine transition in emission to map out neutral hydrogen in large volumes of the universe. Unfortunately, large spatial scales are completely contaminated with spectrally smooth astrophysical foregrounds which are orders of magnitude brighter than the signal. This contamination also leaks into smaller radial and angular modes to form a foreground wedge, further limiting the usefulness of 21-cm observations for different science cases, especially cross-correlations with tracers that have wide kernels in the radial direction. In this paper, we investigate reconstructing these modes within a forward modeling framework. Starting with an initial density field, a suitable bias parameterization and non-linear dynamics to model the observed 21-cm field, our reconstruction proceeds by combining the likelihood of a forward simulation to match the observations (under given modeling error and a data noise model) with the Gaussian prior on initial conditions and maximizing the obtained posterior. For redshifts $z=2$ and $4$, we are able to reconstruct 21cm field with cross correlation, $r_c > 0.8$ on all scales for both our optimistic and pessimistic assumptions about foreground contamination and for different levels of thermal noise. The performance deteriorates slightly at $z=6$. The large-scale line-of-sight modes are reconstructed almost perfectly. We demonstrate how our method also reconstructs baryon acoustic oscillations, outperforming standard methods on all scales. We also describe how our reconstructed field can provide superb clustering redshift estimation at high redshifts, where it is otherwise extremely difficult to obtain dense spectroscopic samples, as well as open up cross-correlation opportunities with projected fields (e.g. lensing) which are restricted to modes transverse to the line of sight.

We present a method to reconstruct the initial conditions of the universe using observed galaxy positions and luminosities under the assumption that the luminosities can be calibrated with weak lensing to give the mean halo mass. Our method relies on following the gradients of forward model and since the standard way to identify halos is non-differentiable and results in a discrete sample of objects, we propose a framework to model the halo position and mass field starting from the non-linear matter field using Neural Networks. We evaluate the performance of our model with multiple metrics. Our model is more than $95\%$ correlated with the halo-mass fields up to $k\sim 0.7 {\rm h/Mpc}$ and significantly reduces the stochasticity over the Poisson shot noise. We develop a data likelihood model that takes our modeling error and intrinsic scatter in the halo mass-light relation into account and show that a displaced log-normal model is a good approximation to it. We optimize over the corresponding loss function to reconstruct the initial density field and develop an annealing procedure to speed up and improve the convergence. We apply the method to halo number densities of $\bar{n} = 2.5\times 10^{-4} -10^{-3}({\rm h/Mpc})^3$, typical of current and future redshift surveys, and recover a Gaussian initial density field, mapping all the higher order information in the data into the power spectrum. We show that our reconstruction improves over the standard reconstruction. For baryonic acoustic oscillations (BAO) the gains are relatively modest because BAO is dominated by large scales where standard reconstruction suffices. We improve upon it by $\sim 15-20\%$ in terms of error on BAO peak as estimated by Fisher analysis at $z=0$. We expect larger gains will be achieved when applying this method to the broadband linear power spectrum reconstruction on smaller scales.

One of the main unsolved problems of cosmology is how to maximize the extraction of information from nonlinear data. If the data are nonlinear the usual approach is to employ a sequence of statistics (N-point statistics, counting statistics of clusters, density peaks or voids etc.), along with the corresponding covariance matrices. However, this approach is computationally prohibitive and has not been shown to be exhaustive in terms of information content. Here we instead develop a Bayesian approach, expanding the likelihood around the maximum posterior of linear modes, which we solve for using optimization methods. By integrating out the modes using perturbative expansion of the likelihood we construct an initial power spectrum estimator, which for a fixed forward model contains all the cosmological information if the initial modes are gaussian distributed. We develop a method to construct the window and covariance matrix such that the estimator is explicitly unbiased and nearly optimal. We then generalize the method to include the forward model parameters, including cosmological and nuisance parameters, and primordial non-gaussianity. We apply the method in the simplified context of nonlinear structure formation, using either simplified 2-LPT dynamics or N-body simulations as the nonlinear mapping between linear and nonlinear density, and 2-LPT dynamics in the optimization steps used to reconstruct the initial density modes. We demonstrate that the method gives an unbiased estimator of the initial power spectrum, providing among other a near optimal reconstruction of linear baryonic acoustic oscillations.

We present nbodykit, an open-source, massively parallel Python toolkit for analyzing large-scale structure (LSS) data. Using Python bindings of the Message Passing Interface (MPI), we provide parallel implementations of many commonly used algorithms in LSS. nbodykit is both an interactive and scalable piece of scientific software, performing well in a supercomputing environment while still taking advantage of the interactive tools provided by the Python ecosystem. Existing functionality includes estimators of the power spectrum, 2 and 3-point correlation functions, a Friends-of-Friends grouping algorithm, mock catalog creation via the halo occupation distribution technique, and approximate N-body simulations via the FastPM scheme. The package also provides a set of distributed data containers, insulated from the algorithms themselves, that enable nbodykit to provide a unified treatment of both simulation and observational data sets. nbodykit can be easily deployed in a high performance computing environment, overcoming some of the traditional difficulties of using Python on supercomputers. We provide performance benchmarks illustrating the scalability of the software. The modular, component-based approach of nbodykit allows researchers to easily build complex applications using its tools. The package is extensively documented at http://nbodykit.readthedocs.io, which also includes an interactive set of example recipes for new users to explore. As open-source software, we hope nbodykit provides a common framework for the community to use and develop in confronting the analysis challenges of future LSS surveys.

We present several methods to accurately estimate Lagrangian bias parameters and substantiate them using simulations. In particular, we focus on the quadratic terms, both the local and the non local ones, and show the first clear evidence for the latter in the simulations. Using Fourier space correlations, we also show for the first time, the scale dependence of the quadratic and non-local bias coefficients. For the linear bias, we fit for the scale dependence and demonstrate the validity of a consistency relation between linear bias parameters. Furthermore we employ real space estimators, using both cross-correlations and the Peak-Background Split argument. This is the first time the latter is used to measure anisotropic bias coefficients. We find good agreement for all the parameters among these different methods, and also good agreement for local bias with ESP$\tau$ theory predictions. We also try to exploit possible relations among the different bias parameters. Finally, we show how including higher order bias reduces the magnitude and scale dependence of stochasticity of the halo field.

A new generation of surveys will soon map large fractions of sky to ever greater depths and their science goals can be enhanced by exploiting cross correlations between them. In this paper we study cross correlations between the lensing of the CMB and biased tracers of large-scale structure at high $z$. We motivate the need for more sophisticated bias models for modeling increasingly biased tracers at these redshifts and propose the use of perturbation theories, specifically Convolution Lagrangian Effective Field Theory ({\sc CLEFT}). Since such signals reside at large scales and redshifts, they can be well described by perturbative approaches. We compare our model with the current approach of using scale independent bias coupled with fitting functions for non-linear matter power spectra, showing that the latter will not be sufficient for upcoming surveys. We illustrate our ideas by estimating $\sigma_8$ from the auto- and cross-spectra of mock surveys, finding that {\sc CLEFT} returns accurate and unbiased results at high $z$. We discuss uncertainties due to the redshift distribution of the tracers, and several avenues for future development.

We describe a simple and fast algorithm for identifying friends-of-friends features and prove its correctness. The algorithm avoids unnecessary expensive neighbor queries, uses minimal memory overhead, and rejects slowdown in high over-density regions. We define our algorithm formally based on pair enumeration, a problem that has been heavily studied in fast 2-point correlation codes and our reference implementation employs a dual KD-tree correlation function code. We construct features in a hierarchical tree structure, and use a splay operation to reduce the average cost of identifying the root of a feature from $O[\log L]$ to $O[1]$ ($L$ is the size of a feature) without additional memory costs. This reduces the overall time complexity of merging trees from $O[L\log L]$ to $O[L]$, reducing the number of operations per splay by orders of magnitude. We next introduce a pruning operation that skips merge operations between two fully self-connected KD-tree nodes. This improves the robustness of the algorithm, reducing the number of merge operations in high density peaks from $O[\delta^2]$ to $O[\delta]$. We show that for cosmological data set the algorithm eliminates more than half of merge operations for typically used linking lengths $b \sim 0.2$ (relative to mean separation). Furthermore, our algorithm is extremely simple and easy to implement on top of an existing pair enumeration code, reusing the optimization effort that has been invested in fast correlation function codes.

We analyse the Baryon Acoustic Oscillation (BAO) signal of the final Baryon Oscillation Spectroscopic Survey (BOSS) data release (DR12). Our analysis is performed in Fourier-space, using the power spectrum monopole and quadrupole. The dataset includes $1\,198\,006$ galaxies over the redshift range $0.2 < z < 0.75$. We divide this dataset into three (overlapping) redshift bins with the effective redshifts $\zeff = 0.38$, $0.51$ and $0.61$. We demonstrate the reliability of our analysis pipeline using N-body simulations as well as $\sim 1000$ MultiDark-Patchy mock catalogues, which mimic the BOSS-DR12 target selection. We apply density field reconstruction to enhance the BAO signal-to-noise ratio. By including the power spectrum quadrupole we can separate the line-of-sight and angular modes, which allows us to constrain the angular diameter distance $D_A(z)$ and the Hubble parameter $H(z)$ separately. We obtain two independent $1.6\%$ and $1.5\%$ constraints on $D_A(z)$ and $2.9\%$ and $2.3\%$ constraints on $H(z)$ for the low ($\zeff=0.38$) and high ($\zeff=0.61$) redshift bin, respectively. We obtain two independent $1\%$ and $0.9\%$ constraints on the angular averaged distance $D_V(z)$, when ignoring the Alcock-Paczynski effect. The detection significance of the BAO signal is of the order of $8\sigma$ (post-reconstruction) for each of the three redshift bins. Our results are in good agreement with the Planck prediction within $\Lambda$CDM. This paper is part of a set that analyses the final galaxy clustering dataset from BOSS. The measurements and likelihoods presented here are combined with others in~\citet{Alam2016} to produce the final cosmological constraints from BOSS.

We investigate the anisotropic clustering of the Baryon Oscillation Spectroscopic Survey (BOSS) Data Release 12 (DR12) sample, which consists of $1\,198\,006$ galaxies in the redshift range $0.2 < z < 0.75$ and a sky coverage of $10\,252\,$deg$^2$. We analyse this dataset in Fourier space, using the power spectrum multipoles to measure Redshift-Space Distortions (RSD) simultaneously with the Alcock-Paczynski (AP) effect and the Baryon Acoustic Oscillation (BAO) scale. We include the power spectrum monopole, quadrupole and hexadecapole in our analysis and compare our measurements with a perturbation theory based model, while properly accounting for the survey window function. To evaluate the reliability of our analysis pipeline we participate in a mock challenge, which resulted in systematic uncertainties significantly smaller than the statistical uncertainties. While the high-redshift constraint on $f\sigma_8$ at $z_{\rm eff}=0.61$ indicates a small ($\sim 1.4\sigma$) deviation from the prediction of the Planck $\Lambda$CDM model, the low-redshift constraint is in good agreement with Planck $\Lambda$CDM. This paper is part of a set that analyses the final galaxy clustering dataset from BOSS. The measurements and likelihoods presented here are combined with others in~\citet{Alam2016} to produce the final cosmological constraints from BOSS.