Normalized to: Workgroup, T.
[1]
oai:arXiv.org:1705.07959 [pdf] - 1583745
Comparison of statistical sampling methods with ScannerBit, the GAMBIT
scanning module
Submitted: 2017-05-22, last modified: 2017-10-15
We introduce ScannerBit, the statistics and sampling module of the public,
open-source global fitting framework GAMBIT. ScannerBit provides a standardised
interface to different sampling algorithms, enabling the use and comparison of
multiple computational methods for inferring profile likelihoods, Bayesian
posteriors, and other statistical quantities. The current version offers
random, grid, raster, nested sampling, differential evolution, Markov Chain
Monte Carlo (MCMC) and ensemble Monte Carlo samplers. We also announce the
release of a new standalone differential evolution sampler, Diver, and describe
its design, usage and interface to ScannerBit. We subject Diver and three other
samplers (the nested sampler MultiNest, the MCMC GreAT, and the native
ScannerBit implementation of the ensemble Monte Carlo algorithm T-Walk) to a
battery of statistical tests. For this we use a realistic physical likelihood
function, based on the scalar singlet model of dark matter. We examine the
performance of each sampler as a function of its adjustable settings, and the
dimensionality of the sampling problem. We evaluate performance on four
metrics: optimality of the best fit found, completeness in exploring the
best-fit region, number of likelihood evaluations, and total runtime. For
Bayesian posterior estimation at high resolution, T-Walk provides the most
accurate and timely mapping of the full parameter space. For profile likelihood
analysis in less than about ten dimensions, we find that Diver and MultiNest
score similarly in terms of best fit and speed, outperforming GreAT and T-Walk;
in ten or more dimensions, Diver substantially outperforms the other three
samplers on all metrics.