HOCO 2.0

How to read the verification metrics

ROC AUC measures how well the forecast ranks lightning-prone cells above non-lightning cells across every possible decision threshold. A value near 1.00 means the system almost always ranks true lightning environments above quiet environments, while a value near 0.50 behaves like random guessing. Because ROC looks at both hit rate and false-alarm rate over all thresholds, it is a strong measure of overall discrimination skill.

PR-AUC focuses on precision and recall, so it answers a slightly different question: when the model highlights a grid cell as risky, how often is that signal useful, and how many true lightning cells are being captured? This matters a lot for lightning because positive events are relatively rare compared with the total number of grid cells. A model can post a respectable ROC AUC while still having modest precision in rare-event situations, which is why PR-AUC is a valuable companion metric.

Brier score measures the average squared error between the forecast probability and the observed binary outcome. Lower is better, with 0.00 being perfect. It complements ROC/PR by checking whether the probabilities themselves are numerically sensible, not just well ranked.

The reliability curve compares the average forecast probability in each occupied bin with the actual observed lightning frequency in that bin. Curves close to the diagonal are well calibrated. Above the diagonal means HOCO was underconfident there; below it means the forecast probabilities were too high.

The flash-density chart adds intensity context by plotting observed strikes per approximate 1,000 km^2 against forecast-probability bins. HOCO still does not forecast explicit flash counts, so this is a diagnostic relationship rather than a direct count-forecast skill score, but it helps show whether higher probabilities are lining up with more concentrated lightning.

The reference systems below are included as context only. Their domains, lead times, targets (e.g. total lightning vs cloud-to-ground lightning), spatial scales, and verification methods differ from HOCO, so they should not be treated as direct like-for-like rankings against the curves above.

HOCO verification scale: the current verification logic evaluates forecasts on an approximately 8 km grid over the UK and then aggregates neighbouring cells into approximately 24 km spatial blocks before scoring. That reduces the worst storm-clustering problem from treating every adjacent grid cell as fully independent. It is still not a full storm-object verification system, but it now includes both calibration diagnostics and an observed density view in addition to occurrence discrimination.

A block is still treated as a binary event for ROC AUC, PR-AUC, Brier score, and the reliability curve: if at least one lightning strike lands inside it, the observed occurrence label is positive. The added density diagnostic partly addresses that limitation by also tracking how many strikes accumulated within each probability bin, but it should still be read as supporting context rather than a full explicit-intensity forecast.

Forecast polygons are now matched more strictly as well: there is no extra blanket 10 km polygon buffer in the verifier, and overlapping forecast zones are combined into a single block probability instead of stopping at the first matching zone. That makes the benchmark less optimistic than the earlier version.

Research note: public lightning papers do not all publish the same skill metrics. A few provide scalar ROC AUC or PR-AUC values, while others focus on CSI, POD/FAR, reliability, or lead time instead. The table below prioritizes studies that actually report AUC-style values, and leaves the metric cell blank where a directly comparable AUC/PR-AUC was not published in the cited source.