We describe an extension of the Enzo code to enable fully-coupled radiation hydrodynamical simulation of inhomogeneous reionization in large $\sim (100 Mpc)^3$ cosmological volumes with thousands to millions of point sources. We solve all dynamical, radiative transfer, thermal, and ionization processes self-consistently on the same mesh, as opposed to a postprocessing approach which coarse-grains the radiative transfer. We do, however, employ a simple subgrid model for star formation which we calibrate to observations. Radiation transport is done in the grey flux-limited diffusion (FLD) approximation, which is solved by implicit time integration split off from the gas energy and ionization equations, which are solved separately. This results in a faster and more robust scheme for cosmological applications compared to the earlier method. The FLD equation is solved using the hypre optimally scalable geometric multigrid solver from LLNL. By treating the ionizing radiation as a grid field as opposed to rays, our method is scalable with respect to the number of ionizing sources, limited only by the parallel scaling properties of the radiation solver. We test the speed and accuracy of our approach on a number of standard verification and validation tests. We show by direct comparison with Enzo's adaptive ray tracing method Moray that the well-known inability of FLD to cast a shadow behind opaque clouds has a minor effect on the evolution of ionized volume and mass fractions in a reionization simulation validation test. We illustrate an application of our method to the problem of inhomogeneous reionization in a 80 Mpc comoving box resolved with $3200^3$ Eulerian grid cells and dark matter particles.

In this first of several application papers, we investigate the mechanics of reionization from stellar sources in high-z galaxies, the utility of various clumping factors on estimating the recombination time in the IGM, and the photon budget required to achieve reionization. We test the accuracy of the static and time-dependent models of Madau et al. as predictors of reionization completion/maintenance. We simulate a WMAP7 LCDM cosmological model in a 20 Mpc comoving cube with 800^3 uniform fluid cells and dark matter particles. By tuning our star formation to approximately match the observed star formation rate density and luminosity function, we created a fully coupled radiation-hydro realization of H reionization which begins to ionize at z~10 and completes at z~5.8. We find that roughly 2 ionizing photons per H atom are required to convert the neutral IGM to a highly ionized state, which supports the "photon starved" scenario discussed by Bolton & Haehnelt. The events during reionization that lead to this number can generally be described as inside-out, but in reality the narrative depends on the level of ionization of the gas one attributes to as ionized. We find that the formula for the UV photon production rate dN/dt_ion(z) needed to maintain the IGM in an ionized state derived by Madau et al. should not be used to predict the epoch of reionization completion because it ignores history-dependent terms in the global ionization balance which are not ignorable. We find that the time-dependent model for the ionized volume fraction Q_HII is more predictive, but overestimates the redshift of reionization completion by delta_z~1. We propose a revised formulation of the time-dependent model which agrees with our simulation to O(1%). Finally, we use our simulation to estimate a global UV escape fraction due to circumgalactic gas resolved on our mesh to be <f_esc>~0.7.

This paper describes the open-source code Enzo, which uses block-structured adaptive mesh refinement to provide high spatial and temporal resolution for modeling astrophysical fluid flows. The code is Cartesian, can be run in 1, 2, and 3 dimensions, and supports a wide variety of physics including hydrodynamics, ideal and non-ideal magnetohydrodynamics, N-body dynamics (and, more broadly, self-gravity of fluids and particles), primordial gas chemistry, optically-thin radiative cooling of primordial and metal-enriched plasmas (as well as some optically-thick cooling models), radiation transport, cosmological expansion, and models for star formation and feedback in a cosmological context. In addition to explaining the algorithms implemented, we present solutions for a wide range of test problems, demonstrate the code's parallel performance, and discuss the Enzo collaboration's code development methodology.

We describe an extension of the cosmological hydrodynamics code ENZO to include the self-consistent transport of ionizing radiation modeled in the flux-limited diffusion approximation. A novel feature of our algorithm is a coupled implicit solution of radiation transport, ionization kinetics, and gas photoheating, making the timestepping for this portion of the calculation resolution independent. The implicit system is coupled to the explicit cosmological hydrodynamics through operator splitting and solved with scalable multigrid methods. We summarize the numerical method, present a verification test on cosmological Stromgren spheres, and then apply it to the problem of cosmological hydrogen reionization.

We consider a PDE system comprising compressible hydrodynamics, flux-limited diffusion radiation transport and chemical ionization kinetics in a cosmologically-expanding universe. Under an operator-split framework, the cosmological hydrodynamics equations are solved through the Piecewise Parabolic Method, as implemented in the Enzo community hydrodynamics code. The remainder of the model, including radiation transport, chemical ionization kinetics, and gas energy feedback, form a stiff coupled PDE system, which we solve using a fully-implicit inexact Newton approach, and which forms the crux of this paper. The inner linear Newton systems are solved using a Schur complement formulation, and employ a multigrid-preconditioned conjugate gradient solver for the inner Schur systems. We describe this approach and provide results on a suite of test problems, demonstrating its accuracy, robustness, and scalability to very large problems.

The development of radiation hydrodynamical methods that are able to follow gas dynamics and radiative transfer self-consistently is key to the solution of many problems in numerical astrophysics. Such fluid flows are highly complex, rarely allowing even for approximate analytical solutions against which numerical codes can be tested. An alternative validation procedure is to compare different methods against each other on common problems, in order to assess the robustness of the results and establish a range of validity for the methods. Previously, we presented such a comparison for a set of pure radiative transfer tests (i.e. for fixed, non-evolving density fields). This is the second paper of the Cosmological Radiative Transfer (RT) Comparison Project, in which we compare 9 independent RT codes directly coupled to gasdynamics on 3 relatively simple astrophysical hydrodynamics problems: (5) the expansion of an H II region in a uniform medium; (6) an ionization front (I-front) in a 1/r^2 density profile with a flat core, and (7), the photoevaporation of a uniform dense clump. Results show a broad agreement between the different methods and no big failures, indicating that the participating codes have reached a certain level of maturity and reliability. However, many details still do differ, and virtually every code has showed some shortcomings and has disagreed, in one respect or another, with the majority of the results. This underscores the fact that no method is universal and all require careful testing of the particular features which are most relevant to the specific problem at hand.

In this paper we describe our massively parallel version of Enzo, a multiphysics, parallel, AMR application for simulating cosmological structure formation developed at UCSD and Columbia. We describe its physics, numerical algorithms, implementation, and performance on current terascale platforms. We also discuss our future plans and some of the challenges we face as we move to the petascale.