This paper presents CUBEP3M, a publicly-available high performance cosmological N-body code and describes many utilities and extensions that have been added to the standard package. These include a memory-light runtime SO halo finder, a non-Gaussian initial conditions generator, and a system of unique particle identification. CUBEP3M is fast, its accuracy is tuneable to optimize speed or memory, and has been run on more than 27,000 cores, achieving within a factor of two of ideal weak scaling even at this problem size. The code can be run in an extra-lean mode where the peak memory imprint for large runs is as low as 37 bytes per particles, which is almost two times leaner than other widely used N-body codes. However, load imbalances can increase this requirement by a factor of two, such that fast configurations with all the utilities enabled and load imbalances factored in require between 70 and 120 bytes per particles. CUBEP3M is well designed to study large scales cosmological systems, where imbalances are not too large and adaptive time-stepping not essential. It has already been used for a broad number of science applications that require either large samples of non-linear realizations or very large dark matter N-body simulations, including cosmological reionization, halo formation, baryonic acoustic oscillations, weak lensing or non-Gaussian statistics. We discuss the structure, the accuracy, known systematic effects and the scaling performance of the code and its utilities, when applicable.

The redshifted 21-cm line of distant neutral H atoms provides a probe of the cosmic ``dark ages'' and the epoch of reionization (``EOR'') which ended them. The radio continuum produced by this redshifted line can be seen in absorption or emission against the CMB at meterwaves, yielding information about the thermal and ionization history of the universe and the primordial density perturbation spectrum that led to galaxy and large-scale structure formation. Observing this 21-cm background is a great challenge. A new generation of low-frequency radio arrays is currently under development to search for this background. Accurate theoretical predictions of the spectrum and anisotropy of this background, necessary to guide and interpret future observations, are also quite challenging. It is necessary to model the inhomogeneous reionization of the intergalactic medium and determine the spin temperature of the 21-cm transition and its variations in time and space as it decouples from the temperature of the CMB. Here, we focus on just a few of the predictions for the 21-cm background from the EOR, based on our newest, large-scale simulations of patchy reionization. These simulations are the first with enough N-body particles (from 5 to 29 billion) and radiative transfer rays to resolve the formation of and trace the ionizing radiation from each of the millions of dwarf galaxies believed responsible for reionization, down to 10^8 M_solar, in a cubic volume large enough (90 and 163 comoving Mpc on a side) to make meaningful statistical predictions of the fluctuating 21-cm background. (abridged)

The Cosmic Dark Ages and the Epoch of Reionization constitute a crucial missing link in our understanding of the evolution of the intergalactic medium and the formation and evolution of galaxies. Due to the complex nature of this global process it is best studied through large-scale numerical simulations. This presents considerable computational challenges. The dominant contributors of ionizing radiation were dwarf galaxies. These tiny galaxies must be resolved in very large cosmological volumes in order to derive their clustering properties and the corresponding observational signatures correctly, which makes this one of the most challenging problems of numerical cosmology. We have recently performed the largest and most detailed simulations of the formation of early cosmological large-scale structures and their radiative feedback leading to cosmic reionization. This was achieved by running extremely large (up to 29 billion-particle) N-body simulations of the formation of the Cosmic Web, with enough particles and sufficient force resolution to resolve all the galactic halos with total masses larger than 10^8 Solar masses in computational volumes of up to (163 Mpc)^3. These results were then post-processed by propagating the ionizing radiation from all sources by using fast and accurate ray-tracing radiative transfer method. Both of our codes are parallelized using a combination of MPI and OpenMP and to this date have been run efficiently on up to 2048 cores (N-body) and up to 10000 cores (radiative transfer) on the newly-deployed Sun Constellation Linux Cluster at the Texas Advanced Computing Center. In this paper we describe our codes, parallelization strategies, scaling and some preliminary scientific results. (abridged)

We present the first large-scale radiative transfer simulations of cosmic reionization, in a simulation volume of (100/h Mpc)^3, while at the same time capturing the dwarf galaxies which are primarily responsible for reionization. We achieve this by combining the results from extremely large, cosmological, N-body simulations with a new, fast and efficient code for 3D radiative transfer, C^2-Ray. The resulting electron-scattering optical depth is in good agreement with the first-year WMAP polarization data. We show that reionization clearly proceeded in an inside-out fashion, with the high-density regions being ionized earlier, on average, than the voids. Ionization histories of smaller-size (5 to 10 comoving Mpc) subregions exibit a large scatter about the mean and do not describe the global reionization history well. The minimum reliable volume size for such predictions is ~30 Mpc. We derive the power-spectra of the neutral, ionized and total gas density fields and show that there is a significant boost of the density fluctuations in both the neutral and the ionized components relative to the total at arcminute and larger scales. We find two populations of HII regions according to their size, numerous, mid-sized (~10 Mpc) regions and a few, rare, very large regions tens of Mpc in size. We derive the statistical distributions of the ionized fraction and ionized gas density at various scales and for the first time show that both distributions are clearly non-Gaussian. (abridged)

We present a new parallel PM N-body code named PMFAST that is freely available to the public. PMFAST is based on a two-level mesh gravity solver where the gravitational forces are separated into long and short range components. The decomposition scheme minimizes communication costs and allows tolerance for slow networks. The code approaches optimality in several dimensions. The force computations are local and exploit highly optimized vendor FFT libraries. It features minimal memory overhead, with the particle positions and velocities being the main cost. The code features support for distributed and shared memory parallelization through the use of MPI and OpenMP, respectively. The current release version uses two grid levels on a slab decomposition, with periodic boundary conditions for cosmological applications. Open boundary conditions could be added with little computational overhead. We present timing information and results from a recent cosmological production run of the code using a 3712^3 mesh with 6.4 x 10^9 particles. PMFAST is cost-effective, memory-efficient, and is publicly available.