[GRASS-dev] GRASS raster compression & performance

GRASS GIS GRASS GIS developer
1

Hi,

since my office grassdata/ becomes huge, I wonder if I should
change to zlib compression.

From the "variables.html" page I get (btw: no mention of this in

the r.compress manual!):

GRASS_INT_ZLIB
    [libgis]
    if the environment variable GRASS_INT_ZLIB exists, new compressed
rasters will be compressed using zlib instead of RLE compression. Such
rasters will have a compressed value of 2 in the cellhd file.
    Obviously, decompression is controlled by the raster's compressed
value, not the environment variable.

I wonder what the performance penalty of zlib could be. Does
anyone have experience? An advantage of stronger compression
could be that the data traffic on NFS on a cluster becomes less...

Opinions welcome,
Markus

2

(back to an older question)

On Sun, Dec 11, 2011 at 10:39 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

since my office grassdata/ becomes huge, I wonder if I should
change to zlib compression.

From the "variables.html" page I get (btw: no mention of this in
the r.compress manual!):

I have now updated the manual page.

GRASS_INT_ZLIB
    [libgis]
    if the environment variable GRASS_INT_ZLIB exists, new compressed
rasters will be compressed using zlib instead of RLE compression. Such
rasters will have a compressed value of 2 in the cellhd file.
    Obviously, decompression is controlled by the raster's compressed
value, not the environment variable.

I wonder what the performance penalty of zlib could be. Does
anyone have experience? An advantage of stronger compression
could be that the data traffic on NFS on a cluster becomes less...

Opinions welcome,
Markus

Meanwhile I discovered the .xz file format:
http://en.wikipedia.org/wiki/Xz
http://tukaani.org/xz/format.html

Maybe worth to be implemented in GRASS?

Markus

3

Markus Neteler wrote:

Meanwhile I discovered the .xz file format:
http://en.wikipedia.org/wiki/Xz
http://tukaani.org/xz/format.html

Maybe worth to be implemented in GRASS?

Container formats such as .gz or .xz aren't relevant.

We could offer LZMA compression (the algorithm used for .xz files) as
an alternative to deflate (zlib), but LZMA is much slower.

There are some benchmarks on the XZ site:

  http://tukaani.org/lzma/benchmarks.html

If anything, I would expect that the difference in file size would be
less for GRASS rasters than for typical files because of the small
chunk sizes (each row is compressed separately).

--
Glynn Clements <glynn@gclements.plus.com>