3
$\begingroup$

I am analyzing ERA5 climate data spanning from 1950 to 2023, covering 73 years. The dataset has a daily temporal resolution and a spatial resolution of 0.25° (1440 x 720).

Each yearly data is stored in a NetCDF file, containing approximately 365 layers.

Attempting to combine these into a single stack results in around 27000 layers, which is overwhelming for my computer, and even the workplace cluster struggles with this volume.

As I'm new to handling large-scale datasets, I'm seeking advice on best practices and effective solutions for managing and processing this type of data.

Note: I have a preference for working with dataframes over NetCDF files as I will perform time series analysis.

Note 2: different types of analysis will be performed in the future. It's very wide: from simple statistics, to indices computations etc. On a grid cell level. So far, I am handling precipitation Prec and PET data to calculate a cumulative water deficit for each grid cell, followed by identifying the maximum deficit values annually, then conducting an extreme value analysis. But I would do other type of analysis in the future.

Improve this question
$\endgroup$
4
  • $\begingroup$ Do you plan to do time series analysis for a certain horizontal location or certain vertical layer? It would be important to read only parts of the data at once and divide the analysis into separate steps. E.g. read only data at one grid point (or a set of grid points) over the whole time span and do the analysis for this grid point (s). Then, this could be repeated for other grid point. $\endgroup$
    – daniel.heydebreck
    Nov 21 at 7:37
  • $\begingroup$ Warning: consider the "chunking" of data, which means in which pattern they are stored. Commonly, the horizontal data of one vertical layer of one time stamp is stored in a row. Then the whole data of the next vertical layer (same time stamp) is stored ... and, then, the next vertical layer ... . After the last layer of time stamp n, the first layer of time stamp n+1 is stored. This is means that accessing all time stamps of at one spatial location is very inefficient and slow. If you wish to do temporal analysis on your data, you should "re-chunk" your data (change the storage pattern). $\endgroup$
    – daniel.heydebreck
    Nov 21 at 7:40
  • $\begingroup$ I'm not an R user, but it sounds like you're looking for a combination of lazy loading and possibly re-chunking the original data, which can change how lazy the loading can be. Also, dataframes and netCDF aren't alternatives to each other; the former is in-memory storage and the latter is on-disk storage. $\endgroup$
    – Deditos
    Nov 25 at 11:36
  • $\begingroup$ @daniel.heydebreck thank you very much. Yes, I aim to perform time series analysis. The thing is I would need to map or analysis specific regions. Could you redirect me to practicals in R referring to what you suggested? $\endgroup$
    – Shunrei
    Dec 1 at 15:29

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.