Can any one help me to handle huge netcdf files each of 1gb memory in loop and at least 2 files at a time in ncl or linux or python or matlab.
for example era_interim daily pressure level datasets.
Earth Science Stack Exchange is a question and answer site for those interested in the geology, meteorology, oceanography, and environmental sciences. It only takes a minute to sign up.Sign up to join this community
For processing large data it is a good practice to call data into RAM in slices(Either by spliting time axis or spatial domain). In the interest of earth sciences python packages Xarray, iris, netCDF4 and h5py are few of the great tools for handling huge hierarchical data. For handling data in a labeled fashion Xarray and Iris will be useful while netCDf4 and h5py are good to process in a gridded way.
My personal suggestion is h5py which is meant for processing and archiving large datasets. Documentation here explains it. If you have netcdf files and want them in hdf5 format a question at Stack Overflow might help.
One option is to not load the entire file at a time. You can use ncgeodataset. The routine allows for the extraction of a subset of data without having to load the entire file or even an array into Matlab. It is great for large datasets.