How accurate is the National Weather Service 8 to 14 Day Outlook?

I know that the forecast accuracy for individual days decreases exponentially past 7 days. But what about the ensemble forecasts that give you the general trend for the week after next? https://www.cpc.ncep.noaa.gov/products/predictions/814day/interactive/index.php

For example, the 8 to 14 day forecast today shows Madison significantly above Normal:

What are the chances that the temperatures for the next week will actually will be below the 50th percentile for that time of year?

Bonus: Is there a chart showing how much 8 to 14 day weather forecasting has improved over the years?

• Dec 10, 2021 at 0:27

To start with, quite a bit of the odds is actually built into the product. They don't include the scale for the values right on your interactive link, but they do show it as percentage odds on the static page:

But the interactive page actually gives a more precise breakdown of the forecasted potentials for a location of interest. From your link, click on the city of interest on the map or put it in the "Find address or place" box, and you get:

The green circled section at left is their display of odds for their three categorical options. Breaking down those options... isn't as well explained as it could be. They don't give value ranges for what each category means, and it took a little wandering to find their mathematical description of what constitutes above/near/below normal, using the standard distribution calculator from here (which matches this CPC page) and combining with this climatology plotter, it looks like the forecast roughly is saying the chances of daily average temperature ranges are: | temperature range for daily average| odds | |:--- |:---:| | >25 °F | 45% | | 14-25 °F | 33% | | <14 °F | 22% | (I'm just looking at average temperature for each day, which is the average of the high and low temperature each day... but the same idea applies to the highs and lows... being within about 5 degrees of average is considered near normal this time of year based on their method)

So right up front, their forecast isn't saying anything too emphatic or perhaps too useful to many. It shows that the forecast leans warm, but not even half of the outcomes will wind up quite warm (more than 5 degrees above normal). Put another another way, it's like saying there's a 45% chance of heavy rain, a 33% chance of fog and drizzle, and a 22% chance of being sunny. Will your event be rained out? Hard to have too much confidence! But yet it does at least show the lean (for my example, it does seem weather is at least be a factor... for this, same idea: there's only a bit of lean towards warm, but the odds of it being abnormally cold look a fair bit lower)

This doesn't directly allow us to calculate the odds of the results coming in anywhere below average as you've asked for... but it would seem a fair estimate is cutting the near normal group up with the same scaling as the below-to-above category ratio, which would give this site a 33% chance of winding up below the 50th percentile.

Now that is only the forecast, based generally on model ensembles and such. It still doesn't include any measure of how *reliable* the forecasts are. The Climate Prediction Center site itself actually has some info towards that aspect of your question... unfortunately the page you happened to come from them had bad verification links. But the main [CPC page][7] (as of now at least) has a [proper link to verification][7], on the left sidebar under Outlooks(Forecast) > Verification.

Then we need to trudge through the complexity of how forecasts are scored, particularly areal ones. The CPC verification focuses around a Heidke score.

If I understand everything, the basics are: 0 is no skill, 100 is perfect. Or 0 bad, 100 great. Here are those Heidke Skill Scores for the 8-14 day temperature forecast over the past year as of yesterday:

(These are just focusing on the three color category types on the map, the orange\red region, the gray region, and the blue region...)

So I believe that says a mean skill score around 20 suggests a forecast only ends up having its skill about 1 out of 5 times. So basically that knocks 80% off the indicated category differences the models suggested... such that it's really more like a 45% chance ($$.50-\frac{.33}{5}$$) that it comes in under the mean, a 55% chance it comes in over the mean.

So for all these graphics are shown in like the media, they really aren't saying much. Furthermore, would think that the lighter orange areas tend to have less categorical skill than the deeper red areas because errors are often geographic (due to systems being slower/faster/deeper/shallower than forecast), meaning the border areas are more likely to swing into the opposite category... so the skill is probably even less yet. It's literally a forecast of a few percent improvement on even... something like a 53% chance a coin comes up heads and 47% chance it comes up tails for yours, pretty unhelpful knowledge for any specific flip.

Only if the tiny return gained from using such products is scaled to big-value repeatable things is their significant benefit... namely things like agriculture... where the small percentage gained is multiplied by a lot of money/food, which over the long-run will equate to tangible gains of expected value return.

How have these forecasts done in the long range? The main verification page only shows skill score for the 8-14 day over the past year. I believe this page may have longer-term plots, as I'm pretty sure I've seen them in the past... but the page is down right now.

The Heilke graph above is expectedly noisy but roughly steady; there haven't been drastic 8-14 day forecast tool changes in the past year. All I can find right now is graphs of monthly and seasonal forecasts over the longer-run. Here's the one-month forecast skill variation:

Not a lot of change, nor a lot of skill.

Likewise this journal article and sites like this one shows there hasn't been much improvement in seasonal scores in general.

From everything I've seen, well-made longer-range forecasts, whether for two weeks or seasonal temperatures or hurricanes or whatever, they all tend to have "skill"... as in they are the slightest bit right more than they're wrong. But in terms of making plans around, they typically don't say too much a person can use at all. And to be "skillful" they also have to have very broad range categories (a forecast of 16 named tropical cyclones is almost certain to be wrong, but a range of 13-20 named tropical cyclones like NOAA gave this year is more likely to be right... even if it does cover everything from a slightly below average season to a near record season [it was still wrong this year!]... likewise an Accuweather forecast of a high of 27 °F three weeks from Thursday is absolutely silly, but a 37% chance the temperature that week will be in the upper third tertile is in theory a valid "gain"... a very very tiny-use one)

For the 8-14 day forecast, unless you're in one of the very high colors on the 8-14 day outlook scale (the very dark blues or deep reds), I wouldn't put much thought towards them at all. They may be a hint that a front (or warmup) may happen at some point during/near that period, but it's hard to narrow down that far into the future as of now. I am convinced such long-range forecast are focused upon far too much versus their typically minuscule usefulness.

• The NOAA pages can be notoriously inconsistent on sidebar links due to inconsistent upgrades. That's why your link had dead verification links. And no idea if the vwt.ncep.noaa.gov (linked from the main verification page as the "interactive web tool") is going to prove useful or not. Dec 10, 2021 at 12:25
• But feel free to nudge me somewhere down the line to see if I can find more longterm data on the 8-14 history specifically, there's probably something out there if enough looking is done, and it may be interesting and potentially may show some improvement given part of the window is on the edge of the forecast window that has improved by model resolution improvement. Dec 10, 2021 at 12:26
• (I assumed you were talking about Madison, WI, let me know if elsewhere..... also I've seen a few graphics showing the peak skill may be nearer the desert southwest, whereas a bit lower up in the northern US) Dec 11, 2021 at 16:13
• Thank you, that was very helpful. Dec 12, 2021 at 0:20
• You're welcome, good question, feel free to pose such questions anytime of course. And very interesting Linux tools you have made, they may be of use to me, so all have gained :-) Dec 12, 2021 at 9:11