I am analyzing ERA5 climate data spanning from 1950 to 2023, covering 73 years. The dataset has a daily temporal resolution and a spatial resolution of 0.25° (1440 x 720).
Each yearly data is stored in a NetCDF file, containing approximately 365 layers.
Attempting to combine these into a single stack results in around 27000 layers, which is overwhelming for my computer, and even the workplace cluster struggles with this volume.
As I'm new to handling large-scale datasets, I'm seeking advice on best practices and effective solutions for managing and processing this type of data.
Note: I have a preference for working with dataframes over NetCDF files as I will perform time series analysis.
Note 2: different types of analysis will be performed in the future. It's very wide: from simple statistics, to indices computations etc. On a grid cell level. So far, I am handling precipitation Prec and PET data to calculate a cumulative water deficit for each grid cell, followed by identifying the maximum deficit values annually, then conducting an extreme value analysis. But I would do other type of analysis in the future.