We review two decades of progress using the Enzo hydrodynamic cosmology code to simulate the Cosmic Dawn, a period of roughly 1 billion years beginning with the formation of the first stars in the universe, and ending with cosmic reionization. Using simulations of increasing size and complexity, working up in length and mass scale and to lower redshifts, a connected narrative is built up covering the entire epoch. In the first part of the paper, we draw on results we and our collaborators have achieved using the Enzo cosmological adaptive mesh refinement code. Topics include the formation of Population III stars, the transition to Population II star formation, chemical enrichment, the assembly of the first galaxies, their high redshift galaxy statistics, and their role in reionization. In the second part of the paper we highlight physical difficulties that will require new, more physically complex simulations to address, drawing from a broader literature survey. We discuss the healthy interplay between self-consistent numerical simulations and analytic and semi-analytic approaches. Finally, we discuss technical advances in hardware and software that will enable a new class of more realistic simulations to be carried out on exascale supercomputers in the future.

Astrophysical and cosmological phenomena involve a large variety of physical processes, and can encompass an enormous range of scales. To effectively investigate these phenomena computationally, applications must similarly support modeling these phenomena on enormous ranges of scales; furthermore, they must do so efficiently on high-performance computing platforms of ever-increasing parallelism and complexity. We describe Enzo-P, a Petascale redesign of the ENZO adaptive mesh refinement astrophysics and cosmology application, along with Cello, a reusable and scalable adaptive mesh refinement software framework, on which Enzo-P is based. Cello's scalability is enabled by the Charm++ Parallel Programming System, whose data-driven asynchronous execution model is ideal for taking advantage of the available but irregular parallelism in adaptive mesh refinement-based applications. We present scaling results on the NSF Blue Waters supercomputer, and outline our future plans to bring Enzo-P to the Exascale Era by targeting highly-heterogeneous accelerator-based platforms.

This paper describes the open-source code Enzo, which uses block-structured adaptive mesh refinement to provide high spatial and temporal resolution for modeling astrophysical fluid flows. The code is Cartesian, can be run in 1, 2, and 3 dimensions, and supports a wide variety of physics including hydrodynamics, ideal and non-ideal magnetohydrodynamics, N-body dynamics (and, more broadly, self-gravity of fluids and particles), primordial gas chemistry, optically-thin radiative cooling of primordial and metal-enriched plasmas (as well as some optically-thick cooling models), radiation transport, cosmological expansion, and models for star formation and feedback in a cosmological context. In addition to explaining the algorithms implemented, we present solutions for a wide range of test problems, demonstrate the code's parallel performance, and discuss the Enzo collaboration's code development methodology.

We study the ionization and thermal evolution of the intergalactic medium during the epoch of \heii reionization by means of radiation hydrodynamical cosmological simulations. We post-process baryonic density fields from a standard optically-thin IGM simulation with a homogeneous galaxy-dominated UV background (UVB) which reionizes \hi and \hei at z=6.5 but does not have any contribution to the ionization of \heii. Quasars with luminosities proportional to the mass of the host halos are then introduced as point sources throughout the 100 Mpc simulation volume consistent with the Pei luminosity function. We evolve the spatial distribution of the \heii ionizing radiation field using a time-implicit variable tensor Eddington factor radiative transfer scheme. Simultaneously, we also solve for the local ionization of \heii to \heii and the associated photoheating of the gas. We find that the percolation of the \heiii regions is essentially complete by z=2.5. When comparing to a self-consistent optically thin simulation we find that in optically thick calculation the gas temperature is higher by a factor of approximately 1.7 at the mean gas density level. We use 300 random lines of sight to compute at $\bar{z} = 2.5 \pm 0.1$ a mean \heii \lya line transmission of $\bar{F} = 0.304 \pm 0.002$. We compare the broadening width of the \hi and \heii \lya lines to the results from the self-consistent optically thin simulation and find a shift by approximately 1.25 km/s of the b-parameter distribution. Estimating the relative broadening width between the two forests shows that the \heii median b-parameter is about 0.8 times the median \hi broadening width. This implies that the \heii absorbers are physically extended consistent with conclusions from observed lines of sight.

In this paper we describe our massively parallel version of Enzo, a multiphysics, parallel, AMR application for simulating cosmological structure formation developed at UCSD and Columbia. We describe its physics, numerical algorithms, implementation, and performance on current terascale platforms. We also discuss our future plans and some of the challenges we face as we move to the petascale.

This paper describes ZEUS-MP, a multi-physics, massively parallel, message- passing implementation of the ZEUS code. ZEUS-MP differs significantly from the ZEUS-2D code, the ZEUS-3D code, and an early "version 1" of ZEUS-MP distributed publicly in 1999. ZEUS-MP offers an MHD algorithm better suited for multidimensional flows than the ZEUS-2D module by virtue of modifications to the Method of Characteristics scheme first suggested by Hawley and Stone (1995), and is shown to compare quite favorably to the TVD scheme described by Ryu et. al (1998). ZEUS-MP is the first publicly-available ZEUS code to allow the advection of multiple chemical (or nuclear) species. Radiation hydrodynamic simulations are enabled via an implicit flux-limited radiation diffusion (FLD) module. The hydrodynamic, MHD, and FLD modules may be used in one, two, or three space dimensions. Self gravity may be included either through the assumption of a GM/r potential or a solution of Poisson's equation using one of three linear solver packages (conjugate-gradient, multigrid, and FFT) provided for that purpose. Point-mass potentials are also supported. Because ZEUS-MP is designed for simulations on parallel computing platforms, considerable attention is paid to the parallel performance characteristics of each module. Strong-scaling tests involving pure hydrodynamics (with and without self-gravity), MHD, and RHD are performed in which large problems (256^3 zones) are distributed among as many as 1024 processors of an IBM SP3. Parallel efficiency is a strong function of the amount of communication required between processors in a given algorithm, but all modules are shown to scale well on up to 1024 processors for the chosen fixed problem size.

In this paper we introduce Enzo, a 3D MPI-parallel Eulerian block-structured adaptive mesh refinement cosmology code. Enzo is designed to simulate cosmological structure formation, but can also be used to simulate a wide range of astrophysical situations. Enzo solves dark matter N-body dynamics using the particle-mesh technique. The Poisson equation is solved using a combination of fast fourier transform (on a periodic root grid) and multigrid techniques (on non-periodic subgrids). Euler's equations of hydrodynamics are solved using a modified version of the piecewise parabolic method. Several additional physics packages are implemented in the code, including several varieties of radiative cooling, a metagalactic ultraviolet background, and prescriptions for star formation and feedback. We also show results illustrating properties of the adaptive mesh portion of the code. Enzo is publicly available and can be downloaded at http://cosmos.ucsd.edu/enzo/ .