3
$\begingroup$

Immediately below is an illustration of my first/prototype attempt to visualize this, which is further explained after that... enter image description here What you're looking at is a year-by-year animation of TMAX (daily max temp) data from the USW00094728.dly file (station at Central Park in New York City) openly available at ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily (see the readme.txt file in that directory for file format info, etc).

The left-hand-side of each frame is Jan1 TMAX and the right-hand-side is Dec31, whereby each frame represents a full year. And the years are 1870 through 2009, each year displayed for 0.2seconds. Two blank frames signal when the animation repeats. The temp scale is $40^o\mbox{C}$ max, $-10^o\mbox{C}$ min.

Also, the daily TMAX data in the displayed animation has been smoothed, i.e., each day denotes a seven-day average including the three days before and after it. Without that smoothing, it's really, really jumpy/discontinuous.

Nevertheless, even with smoothing, there's no TMAX change discernible to the casual observer during the 140 years represented. Of course, with an expected $1$–$2^o$ change on the $40^o$ plot, that might just be too hard to casually notice.

So several ideas occurred to me. (a) Maybe subtract out the daily means (averaged over the 140 years), and display a $\sim10^o$ plot, whose higher resolution would make the small delta's more easily observable. (b) The seven-day smoothing, separately over each year, may not be enough since the year-to-year data is still very jumpy. So maybe additionally smooth by averaging the same day over several before-and-after years. But then that may just be too much unjustifiable data massaging.

So, the overall question here is: what-to-display/how-to-display-it so that the ~150-year TMAX change is easily visible and easily explainable/understandable to the casual observer. Any average person-in-the-street should just be able to look at it and immediately see what's happening, without any gobbledygook/gibberish technical explanation necessary. Also, a side-question: it took a while to find that noaa ghcn data, which seems terrific. Any equally-good or better openly available datasets around?


   EDIT
-----------
Thanks for the comments, @JeopardyTempest and @CamiloRada.

JT's "nighttime low" suggestion (TMIN element from ghcn data) is easy to exhibit, just requiring different command-line arguments to exactly the same program (no packages, by the way, all straight C programming, which I mention after noticing "Programmer" in your profile). So I've illustrated that TMIN animation below (scale, etc, identical to TMAX animation above). However, as pretty much expected, still no easily-visualizable change over the 140 years.

We'll have to try "using anomalies", quoting CR, and hope for better results. And that's probably a reasonable hope, based on your youtube link, which is pretty much exactly what I think I'd be generating, except with Jan,...,Dec along the x-axis rather than the "circle graph" displayed there. (By the way, while it's easy to interpret that circle graph, I think maybe the average person-in-the-street might more easily recognize the left-to-right months-along-the-x-axis display, which is probably what most people are already more familiar with.)

And I'll also take a closer look at that HadCRUT4.4 data used by the youtube's homepage http://www.climate-lab-book.ac.uk/2016/spiralling-global-temperatures/ Google hadn't coughed up that data during the searches I'd tried, and I imagine there's lots more good&interesting stuff I've overlooked.

Meanwhile, immediatly below is the TMIN graph suggested by JT. The "anomaly graphs" will be easy to animate after I code the across-the-years data "select"ion (i.e., analogous to a relational table) from the ghcn data. That'll take a bit of doing (in my available spare time), and I'll try to post some better temperature-increase visualizations when it's done. JT's more elaborate display suggestions will take a bit more elaborate thinking and doing (moreso on the display side than on the computational side), which I'll try to plan/design as I'm coding the more straightforward "anomaly" stuff. Thanks, again, JT and CR.

enter image description here

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ What might be neat is to lay down pixels from all the previous years' highs (using transparency) (with brighter and brighter colors in spots that repeat more and more). That might just show something unique. Certainly you somehow need to incorporate the average, as year to year fluctuations will be larger than longterm changes. That'd seems a unique way of doing so, but no idea if it's easy to do with the packages you're using... nor if it'd show a ton. But seems the more you smooth away real fluctuations, the more you lose in a sense. This would instead use them to make any changes apparent. $\endgroup$
    – JeopardyTempest
    Sep 6, 2018 at 15:44
  • 1
    $\begingroup$ One key thing though may be that the majority of measured warming at most locations is actually in nighttime low temperatures. I wouldn't be surprised if daytime warming were like half a degree or less (no idea). That'd definitely be awful hard to make out in most plotting methods. $\endgroup$
    – JeopardyTempest
    Sep 6, 2018 at 15:45
  • 1
    $\begingroup$ The key is to somehow keep the line of previous years, or at least the year mean. Using anomalies would definitely improve the visualization, but instead of computing anomalies relative to the mean of the whole data set I would recommend to compute them against the pre-1900 or pre-1920 mean. This is a good visualisation that applies to your case youtube.com/watch?v=wXrYvd-LBu0 $\endgroup$
    – Camilo Rada
    Sep 6, 2018 at 17:07

2 Answers 2

Reset to default
2
$\begingroup$

Not really an answer, per se. However, as per JT's comment "you somehow need to incorporate the average" (which mirrors my own speculation in the original question), I've incorporated (subtracted out) the means, and that indeed exhibits a easily-seen trend in the graphs. So it's somewhat of an "answer" to that extent, anyway. And I've also created two new types of graphs, illustrated and discussed (all three types) below, along with some additional questions suggested by them.

First, a static graph of year-by-year, 1870-2009,   $T_{MAX}$ deviations from the mean. Each yearly deviation was calculated as a 365/366-day average of daily deviations, i.e., $\frac1{365(or 366)}\sum_{i=jan1}^{dec31}T_i-T^{mean}_i$,   where $T^{mean}_i$ is the mean $T$ for that particular day-of-year (averaged over all the years, not yet accounting for CR's comment about pre-1900 or pre-1920 means). And each graph's scale is now $\pm4^oC$. And, again, the leftmost point represents 1870's $T-T^{mean}$, the rightmost point 2009's. enter image description here The above graph has no day-by-day smoothing as discussed previously (we're already averaging over all days, anyway), but it does have a 4-year smoothing, i.e., each $T_i$ in the sum is actually a nine-point moving average of the $T$ for that year and day, along with $T$'s for the same day of the previous and next four years. Without any smoothing at all, it looks like this enter image description here So some kind of smoothing seems needed, at least for plots intended for a general audience. I could easily weight each day (e.g., like a normal distribution), assigning the largest weight to the current year's day, but haven't tried any of that yet. Is there any generally-accepted weighting procedure for this kind of data smoothing?

By the way, JT's suggestion to use $T_{MIN}$ actually doesn't seem quite as "monotonic" as the preceding $T_{MAX}$, enter image description here So I'll use $T_{MAX}$'s for the animations below...



The first type of animation is like the one originally illustrated, but now displaying deviations from the mean, whereby the scale's now $\pm4^oC$. So left-to-right is jan1-to-dec31, with the first frame displaying 1870 and the last frame 2009 (with a few blank frames signalling when the animation repeats). To avoid too-"jittery" animations, nine-point moving average smoothing is used "both ways", i.e., first $\pm4$days to get the $T$ for a given day within a year, and then $\pm4$years using the same day-of-year's $T$ to get the displayed $T$. enter image description here So now (unlike the animations in the original question), the displayed $T$-deviations-from-the-mean start out (in 1870) below the baseline, and end up (in 2009) above it, like we'd expect.

But maybe unlike we might expect (at least unlike what I'd expected), watch closely and see that some days of the year seem a bit stubborn about exhibiting higher temps. It's as though climate change has some "national holidays" when it takes a vacation. So that suggested another type of animation to see whether or not that's a real effect.

This one just takes "slices" from the overall dataset along the other axis. That is, left-to-right is now 1870-to-2009, and the animation has 365 frames, the first for jan1 and the last for dec31 (the extra leap-year day omitted). And smoothing's the same as above. enter image description here So this does seem a bit unexpected to me. Each frame, taken by itself, graphs the year-to-year $T$-deviation for a given day. And each of those graphs looks too continuous. Randomly consider, say, sept28. If it happens to be warmer-than-average one year, you wouldn't expect that fact to have much correlation with whether or not next year's sept28 is also warmer. But the graphs typically pretty long blocks of warmer or cooler years.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Is it easy to somehow overlay the year as a counter, so we can tell where it's at in its cycle before the end? $\endgroup$
    – JeopardyTempest
    Sep 8, 2018 at 7:45
  • $\begingroup$ @JeopardyTempest The first animation's replaced with a year-labelled one, as requested (and bumping up against imgur's 2MB limit). By the way, I think it's just my smoothing that accounts for the "unexpected" behavior of the second animation -- don't know why that didn't occur to me at the time. $\endgroup$
    – John Forkosh
    Sep 8, 2018 at 12:06
  • $\begingroup$ (Should have also said... perhaps you can add a date counter to the final graph?) $\endgroup$
    – JeopardyTempest
    Sep 8, 2018 at 15:23
  • $\begingroup$ Nice progress. The stuborness of some days to warm up and the continuity of your last plot could both easily be artifacts of your smoothing, where an exceptionally cold day will keep that day's average low for nine frames. You can try a median instead of a mean or weighted mean, that is usually a good approach to noisy data. $\endgroup$
    – Camilo Rada
    Sep 8, 2018 at 15:53
  • 1
    $\begingroup$ @JeopardyTempest By date, you mean "Jan1"..."Dec31", right (i.e., you're reading it right)? Yeah, but that gif's a bit larger to begin with, and goes over 2MB with labels (or I'd already have uploaded it). It'd be trivial to shrink the width,height #pixels to avoid that, but the whole graph's based on my "imaginary anomaly" about year-to-year correlation for the same day-of-year, explained by smoothing (as noticed in my earlier comment as well as CR's comment). So I'd think the animation's not too interesting to begin with -- but am I wrong about that?, i.e., some reason you'd like to see it? $\endgroup$
    – John Forkosh
    Sep 9, 2018 at 6:04
2
$\begingroup$

JT's and CR's suggestions in the comments following the "first answer" are incorporated in the gif's below. The new ones that correspond to the previously-displayed ones are a bit more "jittery" than those previous ones -- they had a little bug, as follows. The "baseline mean" for the year-by-year animations, with jan1-dec31 along the x-axis, is the mean/average for each day summed over all the displayed years. But my summing for that mean took place >>after<< I'd done the (typically 9-point moving average) smoothing of those daily temps. Probably not what we'd want, although you might be able to argue it's a legitimate procedure to "smooth the baseline". In any event, baselines for gif's below display means-before-smoothing, i.e., the raw daily temps are summed over all displayed years, and those generate animations that are a bit more "jittery".

So first, here's the now-labelled jan1-dec31 animation (each frame displaying all years, left-to-right, for one day-of-the-year). The first "4" in the "4x4 smoothing" label refers to the smoothing across days as discussed earlier, and the second "4" refers to the smoothing across years... enter image description here


Secondly, CR's median suggestion didn't really have much effect, one way or the other. Immediately below are the static gif's showing the yearly-deviations-from-the mean/median, 1869(left)-to-2017(right)... enter image description here enter image description here You probably think they're identical. But download both and quickly click between them. The median's "translated upwards" by maybe $1/4^oC$ (I think that's a mean/median skew), and the shape's ever-so-slightly different in a few spots. Go figure. Anyway, here are the corresponding one-year-per-frame (jan1-dec31 left-to-right) animations... enter image description here enter image description here


In case you might want to "play" with these animations, I've uploaded a statically-linked linux executable of the program at
    http://www.forkosh.com/ghcn.exe   (ignore the windows-like .exe, it's linux elf)
and the .dly file I've been using is at
    http://www.forkosh.com/USW00094728.dly  
although you can get that (and about a zillion other .dly files) from the ghcn link in the original question. (The statically-linked ghcn.exe should include all needed libraries/versions not natively installed by your particular linux distribution, but that doesn't always work. But it did work as compiled/linked on my box, and run on my ISP's box.)

It's run from the command line with up to 9 command-line arguments...

/* --- command-line args (all tests) --- */
int   testnum = ( argc>1? atoi(argv[1]) : 1 );
char  *file   = ( argc>2?      argv[2]  : "USW00094728.dly" ); /*CntrlPk*/
int   msglevel= ( argc>3? atoi(argv[3]) : 0 );
  /* --- command-line args for test#2 --- */
  char *types    =( argc>4?      argv[4]  :  "TMAX" ); /*"TMIN","TMAX"*/
  int  years     =( argc>5? atoi(argv[5]) : 1869149 ); /* yyyyNNN */
  int  meankind  =( argc>6? atoi(argv[6]) :       1 ); /*1=avg, 2=median*/
  int  ndlysmooth=( argc>7? atoi(argv[7]) :       2 ); /*smooth dly tmps*/
  int  nylysmooth=( argc>8? atoi(argv[8]) :       2 ); /*smooth yly tmps*/
  double bigf    =( argc>9? atof(argv[9]) :     4.0 ); /*max graph deg C*/

So my typical run would look like...
    ./ghcn.exe 2 USW00094728.dly 999 TMAX 1869149 2 4 4 4
And your first three args, 2 USW00094728.dly 999 should always be identical (except for that .dly file if you download some others). The first 2 for testnum is the only test generating relevant gif's. The others (actually only one other) mainly prints some data for unit-testing the data collection-and-selection functions. And the 999 for msglevel will print just a few lines which you can ignore.

The remaining six args can be varied. You replace my TMAX by TMIN to get corresponding gif's for you-know-what. (I have yet to think about JT's earlier suggestion for one gif incorporating several ghcn element "types".) Trying any other ghcn element "type" (I haven't tested any yet) will use the same $0.1$ scale factor applied to temps (though that's easily made another arg).

The next arg's the date-plus-#years. My 1869149 means start with year 1869 and use 149 years data in all, in this case meaning 1869-2017 inclusive (and that's pretty much the entire .dly file). The 2 after the years is CR's mean/median, with 1=mean, 2=median. The three numbers after that, coincidentally (but not necessarily) the same 4 4 4 in my example, are (first 4) the number-of-days forward-and-backwards for daily smoothing, and (second 4) #days for yearly smoothing. 0 for either/both means no smoothing in that "direction". Finally, the last 4 is the $\pm$degreesC from the baseline displayed on the graphs.

Output is three files: ghcn.gif, ghcn2.gif and (you guessed it) ghcn3.gif. The first is the year-by-year animation, the second is the day-by-day animation, and the last is the static gif displaying year-average deviation from the mean/median, first year to last.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.