How do meteorologists grade their past forecasts?

Question

In every professional field, practitioners may come home at the end of the day and say "Yeah! I knocked it out of the park!", or alternatively "Oh, man. Gotta do better tomorrow." What about meteorologists?

When a meteorologist makes a weather forecast, she will strive to "get it right", so that when the date in question arrives her prediction will have matched the actual temperature, dew point, wind velocity and direction, precipitation, etc. But, when that date arrives, and the actual conditions have been measured and compared to the prediction, how is the prediction graded?

Precipitation seems to be the most problematic aspect, as it's quite nonlinear and predicted statistically. What does it mean to have "gotten it right" by predicting a 30% chance of precipitation? Yes, it rained 30% of the time? Yes, it rained in 30% of the forecast area? Yes, 30% of the public will answer "Yes" if asked whether it rained?

There's complexity in the other metrics as well. For instance, how are errors in the different values compared? Would it be better to get the temperature spot on but the wind velocity 5MPH high, or the wind exactly right but the temperature 5°F high? What about timing: what if that front came through two hours after it was predicted?

Edit: part of the difficulty is getting clarity about the consumer of the weather information. Fred the Farmer wants to know how much rain he can expect in next day or so, but the exact timing probably isn't important. Pete the Party Planner wants to herd his guests into the bar before the front comes through, but if it's going to rain he probably doesn't care whether it drops a quarter inch or a half. And Francine the Fisherman doesn't really care about the precipitation at all (within limits); her world centers around wind and waves.

I may want to post a second question that is explicitly from the consumer viewpoint: given weather data and meteorologists' historical forecasts, how would I choose a meteorologist?

Answer 1

Disclaimer: I am not a meteorologist.

To answer this question, you need to understand how a forecast is obtained. Basically, meteorologists run computer simulations that predict how weather systems might evolve from current measurements. In fact, data assimilation techniques (http://en.wikipedia.org/wiki/Data_assimilation) are commonly used, so that the predictions given by the model depend not only on current observations, but also upon the history of the weather system. As new data comes in, the evolving model is adjusted towards those values - but not forced to match exactly, as that would destroy the historical information.

Any one model is therefore not expected to be entirely accurate. However, if we run many models simultaneously, and include random variations to reflect the chaotic nature of the weather system (and measurement errors), we obtain something called an "ensemble forecast". This is where the 30% chance of rain comes from: 30% of the simulations predict rain in your area, while the others think it will remain dry. Average/minimum/maximum temperatures etc can also easily be extracted from the ensemble.

So, the accuracy of a forecast can be considered in different ways. It is certainly possible to compare the predictions of one ensemble member to observations: this is an integral part of the data assimilation process. However, it is not necessarily a problem if a member performs poorly on one particular occasion. Different types of measurements may be weighted differently during the comparison (so matching rainfall might be deemed 'more important' than matching temperature; this may vary depending on the type of forecast/its intended use). More generally, you have to look at the long-term statistics of the errors in the forecast. So, if you look at several months of data, did it rain on around a third of the days when you predicted a 30% chance of rain? On average, are your temperature predictions reasonably accurate? Making this assessment on a day-by-day basis is unlikely to be in keeping with the statistical basis of modern weather forecasting.

Answer 2

Generally meteorologists use mean bias and mean error metrics to characterize the magnitude of their "success" for things like precipitation and wind speed. Usually a monthly average of performance metrics (or even seasonal) would be used to summarize those results. Also, it's important to keep in mind that for something like rain, they would be more concerned with the total per day (not the exact time that precip occurred). Furthermore, keep in mind most weather performance is assessed at airports, where there are weather towers, so they don't assess their predictions over some large area, because they need instrument observations to do so.