Weather forecasters

Please wait a moment for the data to load.

Note: This is an amateur comparison. Which is not to say that the data are necessarily wrong but that I do not guarantee that they’re not. If you found a mistake, please report it to me and give me a chance to correct my ways before you sue me.

The goal of my little enterprise was to discover which forecasting service most accurately foretells when it is going to rain or snow.

Every morning, I wrote down 20 selected forecasts for Cracow, Poland, for 0 to 6 days ahead, and the actual weather from the day before. Each day was divided into three periods: ±8:00 < morning < ±13:00 < afternoon < ±18:00 < evening < 22:00. Each such period was described as an r for ‘rain’ or s for ‘no rain’. The forecasted triples of r’s and s’s were then compared to the factual ones.

For each forecaster, the mean accuracy and the standard deviation were calculated, and from these two, its score. The idea behind the scoring is that a randomly accurate forecaster is no better than a consistently inaccurate one. So, the score was calculated with the formula: $\sqrt{μ^2 + (1-σ)^2} / \sqrt{2}$, i.e. the Euclidean distance from the point (0,1) which is where the worst possible accuracy and the worst possible standard deviation meet. Thus, the worst achievable score was 0 and the best was 1. See here for illustration.

As control group, I used my own guess based on a short walk on the balcony. The control group is called ‘me, myself & I’.

The forecasters were organized in a league. Every 15 or 16 days (called ‘a season’), I replaced the less accurate half of forecasters with a set of new ones. Only the control group always continued to the next season, regardless of its score. Forecasters which were down for longer than ⅔ of the duration of the season, were disqualified. All forecasters were awarded points in inverse relationship to the place they took. (League summaries (in Polish) can be downloaded here.)

To decide how many forecasters to include from each country, a formula inspired by the one used to distribute of seats in the European Parliament, was used. Namely: ||1 + ln(population) * HDI||, where population is given in millions. Two exceptions were made: for Iceland (the formula returns -0.02) and for Poland which, being the host, was allowed more forecasters than the 4 it deserved. I included all countries I could find enough forecasters from.

Overall, the data was collected over the period of 152 days from 2012.06.14 to 2012.11.06, from 88 forecasters from 25 countries. Given that from each forecaster, up to six forecasts were taken into account every day, the above totals to 20.503 foreacsted days.

Technical note: all calculations were made in R; all charts &c. are generated using Google Chart Tools.