2
$\begingroup$

Can any one help me to handle huge netcdf files each of 1gb memory in loop and at least 2 files at a time in ncl or linux or python or matlab.

for example era_interim daily pressure level datasets.

$\endgroup$
  • 1
    $\begingroup$ maybe a question more appropriate for Stack Overflow. $\endgroup$ – gansub Dec 24 '17 at 16:00
  • $\begingroup$ We have a set of related questions in this site. For instance: earthscience.stackexchange.com/questions/5259/… $\endgroup$ – arkaia Dec 24 '17 at 16:22
  • 1
    $\begingroup$ pretty vague question. "handle" them how? Do you have an error of some type or sample code you are trying to use? $\endgroup$ – farrenthorpe Dec 24 '17 at 16:40
  • 1
    $\begingroup$ What exactly do you want to do? $\endgroup$ – daniel.neumann Dec 24 '17 at 19:44
  • 1
    $\begingroup$ The answers given here are perhaps useful, but if you can be more specific about what's in the file and what you are trying to do, you may get a more specific answer. $\endgroup$ – Semidiurnal Simon Dec 29 '17 at 11:27
2
$\begingroup$

For processing large data it is a good practice to call data into RAM in slices(Either by spliting time axis or spatial domain). In the interest of earth sciences python packages Xarray, iris, netCDF4 and h5py are few of the great tools for handling huge hierarchical data. For handling data in a labeled fashion Xarray and Iris will be useful while netCDf4 and h5py are good to process in a gridded way.

My personal suggestion is h5py which is meant for processing and archiving large datasets. Documentation here explains it. If you have netcdf files and want them in hdf5 format a question at Stack Overflow might help.

$\endgroup$
1
$\begingroup$

One option is to not load the entire file at a time. You can use ncgeodataset. The routine allows for the extraction of a subset of data without having to load the entire file or even an array into Matlab. It is great for large datasets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.