A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier (model) data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve, plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero-intercept unity-slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data-model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth's inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations.

Ensemble modeling of CMEs provides a probabilistic forecast of CME arrival time which includes an estimation of arrival time uncertainty from the spread and distribution of predictions and forecast confidence in the likelihood of CME arrival. The real-time ensemble modeling of CME propagation uses the WSA-ENLIL+Cone model installed at the CCMC and executed in real-time. The current implementation evaluates the sensitivity of WSA-ENLIL+Cone model simulations of CME propagation to initial CME parameters. We discuss the results of real-time ensemble simulations for a total of 35 CME events between January 2013 - July 2014. For the 17 events where the CME was predicted to arrive at Earth, the mean absolute arrival time prediction error was 12.3 hours, which is comparable to the errors reported in other studies. For predictions of CME arrival at Earth the correct rejection rate is 62% and the false-alarm rate is 38%. The arrival time was within the range of the ensemble arrival predictions for 8 out of 17 events. The Brier Score for CME arrival predictions is 0.15 (where 1 is a perfect forecast), indicating that on average, the predicted likelihood of CME arrival is fairly accurate. The reliability of ensemble CME arrival predictions is heavily dependent on the initial distribution of CME input parameters, particularly the median and spread. Preliminary analysis of the probabilistic forecasts suggests undervariability, indicating that these ensembles do not sample a wide enough spread in CME input parameters. Prediction errors can also arise from ambient model parameters, the accuracy of the solar wind background derived from coronal maps, or other model limitations. Finally, predictions of the Kp geomagnetic index differ from observed values by less than one for 11 out of 17 of the ensembles and Kp prediction errors computed from the mean predicted Kp show a mean absolute error of 1.3.