Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. Here we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC's design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

Remarkable observational advances have established a compelling cross-validated model of the Universe. Yet, two key pillars of this model -- dark matter and dark energy -- remain mysterious. Sky surveys that map billions of galaxies to explore the `Dark Universe', demand a corresponding extreme-scale simulation capability; the HACC (Hybrid/Hardware Accelerated Cosmology Code) framework has been designed to deliver this level of performance now, and into the future. With its novel algorithmic structure, HACC allows flexible tuning across diverse architectures, including accelerated and multi-core systems. On the IBM BG/Q, HACC attains unprecedented scalable performance -- currently 13.94 PFlops at 69.2% of peak and 90% parallel efficiency on 1,572,864 cores with an equal number of MPI ranks, and a concurrency of 6.3 million. This level of performance was achieved at extreme problem sizes, including a benchmark run with more than 3.6 trillion particles, significantly larger than any cosmological simulation yet performed.

We present a set of ultra-large particle-mesh simulations of the LyA forest targeted at understanding the imprint of baryon acoustic oscillations (BAO) in the inter-galactic medium. We use 9 dark matter only simulations which can, for the first time, simultaneously resolve the Jeans scale of the intergalactic gas while covering the large volumes required to adequately sample the acoustic feature. Mock absorption spectra are generated using the fluctuating Gunn-Peterson approximation which have approximately correct flux probability density functions (PDFs) and small-scale power spectra. On larger scales there is clear evidence in the redshift space correlation function for an acoustic feature, which matches a linear theory template with constant bias. These spectra, which we make publicly available, can be used to test pipelines, plan future experiments and model various physical effects. As an illustration we discuss the basic properties of the acoustic signal in the forest, the scaling of errors with noise and source number density, modified statistics to treat mean flux evolution and misestimation, and non-gravitational sources such as fluctuations in the photo-ionizing background and temperature fluctuations due to HeII reionization.