# Regarding compression, shuffle filter of netcdf4

• what are the advantages to enable or disable the shuffle,

• what level [0-9] of compression is optimal,

• what's the best choice of chunk size?

I made some pictures on my tablet to explain everything a bit better. Sorry that they are not that nice.

# Shuffle

If we do not us shuffle, we store on value (float, double, integer, short integer, ...) after another. Thus, each variable itself is stored contiguously in memory. The first bit of a second value follows the last bit of the first value. If we use shuffle, the first bit of all variables is stored contiguously in space, followed by the second bit, and so on.

The figure above shows an array of six 8-bit integers (only 8-bit because it was less painting work ;-) ). The vertical black boxes split in 8 compartments represents on 8-bit integer. The orange arrows from left to right indicate how the bits would be stored with shuffling.

Now we assume some example values: We have an array with the values 4, 2, 4, 5, 7, and 6. It is shown in the figure below. The first row shows the decimal representation. The matrix below is the binary representation. Each binary column represents the corresponding decimal number in the first row. The red arrows indicates the normal storage order. The orange arrows indicate the shuffled storage order.

Now we can write down the order of bits in the memory. I wanted to use color. Therefore, you get another picture:

We have a lot of zeros in the beginning of the binary representation of each value (because we have very low values). The shuffling is favorable in this situation because we have a long contiguous memory part filled with the same number/condition (= 0).

If we had no low numbers but higher numbers, (but) which are close to each other, we would not have a long contiguous memory part filled with 0s (zeros) but it might be six 1s and then six 0s. Commonly in netCDF files we have much larger arrays. Lets say a 4-dim array with 100 x 200 x 30 x 365 values, which are quite similar to each other. Then we would have often 2.19 x 109 contiguous 1s or 0s in memory, which is quite nice for compression.

However, if neighboring values differ considerably, shuffling might not be a good idea.

# Chunking

If we have an uncompressed file (= no chunking; or one big chunk), a n-dim array is stored contiguously in memory: at first all values of the first row (first index) of the array are stored contiguously in memory; then we go to the next row (increment second index by one) and write the second row contiguously into the memory; and so on.

We might also start with the last index and do it the other way around ... - depending on the programming language and on the definition of what are rows and columns. But the principle is the same.

The figure below shows an example for a 2-dim variable with one spatial and one temporal dimension. The red arrows indicate how the data is aligned in memory. The black dots are individual values in our 2-dim array.

Now, we want to extract a time slice (all spatial values at one time step) and a space slice (all temporal values at one location) from our data. In the second case we get a time series (could be a time series of model data). In the first case we get a picture of the current situation (could be for a map). The blue and green markings in the figure below indicate the time and space slices.

The difference between time and space slice is the following. The time slice is fast to extract because the values are stored contiguously in memory. The space slices takes more time because only each sixth value is extracted. Is this example the extra work/time spend is low. However, if we have a large 4-dim variable, the time-slice-extraction will be faster than the space-slice-extraction.

When we compress data, we can chunk it. That means, that we split our large array-variable in sub-arrays. For the user it looks the same from the outside. netCDF-4 does chunking automatically, when we compress data but don't define a chunking pattern.

The figure below shows an example, in which our data was split into three chunks. The red arrows indicate the alignment of our values in the memory. This is just an exemplary alignment.

Now, we extract time and space slices again (figure below). Green and blue indicate the slices, again. Here we see that the space-slice will be much easier/faster to extract then before. However, the extraction of the time slice will take more time.

In addition, the compression will differ when you change the chunk sizes. The background is similar as described in the Shuffle section: depending on which values are neighboring each other, the one or the other chunking might be favorable.

Now to your question: what's the best choice of chunk size?

## focus on usage

It depends on your data usage and you access pattern. If you often extract time series at specific locations, it might be favorable to define the chunks in a way that the values corresponding to one location are close to each other aligned in memory.

But be careful, some software like cdo access the netCDF data always in the same way - independent of what you do with it during the processing. Thus, if you don't access the file directly via the C- or Fortran-netCDF4 routines but via some other software, the best chunking pattern might not depend on what you want to do but how the software does it ... . Some software might - for example - always read data along the spatial dimensions first, although you want data along the time dimension.

## focus on compression:

The size of the compressed file varies with the chunking layout. The situation is similar as described in the Shuffle section. If the data are more effective to compress in time dimension, the chunking in my example figure above might be favorable for file size (e.g. high variability in space but low variability in time).

## Chunking summary

The best choice of the chunk size depends on your access pattern, on the software you use, on your data itself, and on your focus on access time vs. storage space.

# Compression level

The choice of the compression level depends on your aim. If you just want to store the data (in an archive or where-ever) and have it as small as possible, you might use compression level 9.

If you want to access data often and fast, you should choose a lower compression level. We use compression levels up to 4. Above that the reduction in file size is very low (for us) so that we do not have a reason to go beyond.

Example: Currently, I work with an ocean model. We have a lot of land in our model domain (roughly 1/3). Compression of level 1 is sufficient to reduce the file size by approximately 95%. However, this is a special case.

# summary:

I tried to give an overview of the features you asked for. The exact choice depends on you particular use case and cannot be generalized. For you, my answer might be too late but I hope that it might help some others. For me, the document you linked helped a lot in the past.

• Thanks. Damn, I had hoped to get the "Silencium" hat ... ;-) Commented Dec 19, 2017 at 21:04