3
$\begingroup$

I have a sea surface temperature time series, and I want to calculate the (Pearson) correlation coefficients among the nodes in the tropics (30.0N to 30S). In the time series, the land information is present as masked. I do not know how to handle masked data in correlation calculation. Please help. The data I used here is in this link https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing

I tried the following code, which did not work:

      temp5 = 'sst.day.mean.1983.nc'
      fh5 = Dataset(temp5, mode = 'r')
      sst5 = fh5.variables['sst'][:365]
      time = fh5.variables['time'][:]
      lat = fh5.variables['lat'][210:510][::35] #tropics latitude
      lon = fh5.variables['lon'][::45]

      mar_05=[]
      for i in range(len(lat)):
        for j in range(len(lon)):
           for m in range(len(lat)):
              for n in range(len(lon)): 
               mar_05.append(np.corrcoef(sst5[59:90,i,j],sst5[59:90, m,n][0,1]))
      df = pd.DataFrame(data = mar_05)
Improve this question
$\endgroup$
5
  • $\begingroup$ I have tried np.ma.corrcoeff instead of np.corrcoef, but still, the correlation list is NAN. Any plausible solution? $\endgroup$
    – Ruby
    Mar 5 at 19:52
  • $\begingroup$ Are any of your data in sst5 NaN values, including the masked elements? $\endgroup$
    – Deditos
    Mar 7 at 9:31
  • $\begingroup$ @Deditos There are no NAN values in the data; only masked values exist. $\endgroup$
    – Ruby
    Mar 7 at 11:48
  • $\begingroup$ @Deditos thanks for the suggestion, but my problem is still there. I used the NOAA SST, and I defined the latitude of interest (lat = fh5.variables['lat'][210:510][::35]). The latitude range is from [-37.375 -28.625 -19.875 -11.125 -2.375 6.375 15.125 23.875 32.625], the latitude of the tropic. Over the latitude range, the land area is masked, and the ocean data are available. I tried calculating all possible correlation values among the nodes, so the nested loop. I also use ma.corrcoef instead np.corrcoef, but the correlation list contains NAN values only. $\endgroup$
    – Ruby
    Mar 8 at 13:12
  • $\begingroup$ Your lines lat= and lon= don't do what you seem to think they do. They don't set which parts of the axes are active/visible when you reference sst5, they just make lists of the lats and lons that you're interested in, but sst5 knows nothing about their contents. I'll edit my answer. $\endgroup$
    – Deditos
    Mar 8 at 16:26

1 Answer 1

Reset to default
1
$\begingroup$

I'm going to hazard a guess that you're using the NOAA OISST data, although I don't think that makes a difference for the answer here.

Note that your loops,

for i in range(len(lat)):
    for j in range(len(lon)):
        for m in range(len(lat)):
            for n in range(len(lon)):

will just generate the indexes,

i = 0, ..., 8
    j = 0, ..., 31
        m = 0, ..., 8
            n = 0, ..., 31

When you then subset the global SST array using sst5[59:90,i,j] and sst5[59:90,m,n] you're not extracting data from the tropical points in your lat and lon variables, you're just extracting data from the corner of the full, global array, i.e., sst5[:, 0:8, 0:31]. You can see this by printing out sst5.shape in your loop, which will show,

...
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
...

This corner is in Antarctica, hence no SST data and numpy.corrcoef will be seeing only missing data values and trying to divide by a zero covariance, yielding NaN values. My Numpy issues the following warning when this happens:

RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]

You could modify your program to subset the sst5 array before the loops:

sst5_subset = sst5[59:90, 210:510:35, ::45]

and then refer to this array inside the loops:

mar_05.append(np.corrcoef(sst5_subset[:,i,j],sst5_subset[:, m,n])[0,1])

But, personally, I would get rid of the loops and do this as a whole-array operation:

sst_array = sst5_subset.reshape(sst5_subset.shape[0], -1).T
corr = np.ma.corrcoef(sst_array)

A quick picture of the output for one point of interest: enter image description here

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ thanks for the convenient reply, and it saved me from the nested loop. Earlier, I calculated the correlation for the globe and did not realize lat = [ ..] and lon = [] did not fulfill the purpose in the nested loop conditions. Thanks for making me realize the mistake I made. $\endgroup$
    – Ruby
    Mar 9 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.