[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

GDAL · November 11, 2015, 2:42pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------
Changes (by mmetz):

* Attachment "giscompress.tar.gz" added.

new files for lib/gis to support different compressors

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 11, 2015, 2:48pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by mmetz):

Replying to [comment:25 mmetz]:
> Replying to [comment:24 wenzeslaus]:
> > I was not able to make BZIP2 work for me. I would say I miss something
like addition to `GISDEPS` but I was not able to fix it (getting undefined
reference to some BZ function).
>
> You need to add $(BZLIB) to GISDEPS in include/Make/Grass.make. I will
update bzipsupport.patch accordingly.

The patches compressors.patch and compressors.patch are also updated to
properly support NULL compression. Data for the null2 file are now
compressed with the same method like actual raster values. Because NULL
data usually occur in long sequences of identical values, the new generic
RLE algorithm I wrote for lib/gis/compress.c seems to perform best
(highest compression, very fast). However, for consistency it would be
better to use the same compressor for both the null bitsream and the
raster data.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:26>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 11, 2015, 6:14pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

Thanks for the patch for BZIP2, I've tried to do it before but I did
something wrong. Now it works and the tests are running:

{{{
GRASS_COMPRESSOR=BZIP2 ./bin.x86_64-unknown-linux-gnu/grass71
/grassdata/nc_basic_spm_grass7/user1 --exec python -m grass.gunittest.main
--location nc_basic_spm_grass7 --location-type nc
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:27>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 11, 2015, 7:58pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

Replying to [comment:26 mmetz]:
> The patches compressors.patch and compressors.patch are also updated to
properly support NULL compression. Data for the null2 file are now
compressed with the same method like actual raster values.

Tested with `GRASS_COMPRESSOR=LZ4`, `GRASS_COMPRESSOR=BZIP2` using the
[comment:27 command above] but as I mentioned here or in #2349, the test
do not cover NULL files well.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:28>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 12, 2015, 9:44am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by glynn):

Replying to [comment:23 mmetz]:

> As before, the rasterlib makes only partial use of the generic
compression methods: no compression and RLE is handled by the rasterlib
internally, and RLE is not supported for fp maps. Creating uncompressed
raster maps has been and should be only possible with
`Rast_open_new_uncompressed()`.

There's no fundamental reason why RLE can't be supported for FP maps; it's
just seldom useful. RLE is most useful for category data where you have
contiguous areas with the same value, and such maps tend to be integer
maps (e.g. you can't create a reclass of a FP map).

No compression used to be a special case because it means that maps don't
have to be written sequentially: 6.x has G_put_map_row_random() which
allows data to be written to an arbitrary row rather than to the "next"
row. 7.x doesn't have an equivalent because nothing actually used it.

In theory, even compressed maps can be written non-sequentially: just
append the data to the end of the file then update the row pointer for the
row. However, overwriting a row will result in the old data taking up
space in the file.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:29>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 19, 2015, 8:44am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

"User" comment: I would suggest to always activate the NULL file
compression by default, anything else would be confusing.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:30>
GRASS GIS <https://grass.osgeo.org>

GDAL · November 20, 2015, 4:47am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

Replying to [comment:26 mmetz]:
> Data for the null2 file are now compressed with the same method like
actual raster values. Because NULL data usually occur in long sequences of
identical values, the new generic RLE algorithm I wrote for
lib/gis/compress.c seems to perform best (highest compression, very fast).
However, for consistency it would be better to use the same compressor for
both the null bitsream and the raster data.

For advanced users it seems that different compression for FCELL+DCELL and
for NULL+CELL would make sense. If I have a MODIS LST for an island, then
I want to use RLE for NULL files but BZIP2 for DCELL.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:31>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 2, 2015, 7:15pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by sprice):

Hey all, just checking in. What is remaining to get this pushed to the
GRASS v7.1 code?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:32>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 3, 2015, 9:48pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

Replying to [comment:32 sprice]:
> Hey all, just checking in. What is remaining to get this pushed to the
GRASS v7.1 code?

For me two things are yet open.

First:

Replying to [comment:30 neteler]:
> "User" comment: I would suggest to always activate the NULL file
compression by default, anything else would be confusing.

--> not yet reflected in the patch (so, the meaning of the var
`GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
`GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
time and would like to get rid of wasting too much disk space.

Second:

We need a patch for GRASS GIS 7.0 for backward compatibility, at least for
the NULL file compression. Which leads to the suggestion that either RLE
or BZIP should be used here since these two compressors are already
present in releasebranch70.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:33>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 5, 2015, 12:20am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by sprice):

For "backward compatibility" do you mean files written by grass71 can be
read by grass70? Or would you like a similar patch to also work with the
grass70 branch?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:34>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 13, 2015, 9:37pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

Sorryfor the delay.

Replying to [comment:34 sprice]:
> For "backward compatibility" do you mean files written by grass71 can be
read by grass70?

Yes, exactly.

> Or would you like a similar patch to also work with the grass70 branch?

Perhaps in future, once we know that the grass71 implementation runs well
on all OS.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:35>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 16, 2015, 10:03am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by mmetz):

Replying to [comment:33 neteler]:
> Replying to [comment:32 sprice]:
> > Hey all, just checking in. What is remaining to get this pushed to the
GRASS v7.1 code?
>
>
> For me two things are yet open.
>
> First:
>
> Replying to [comment:30 neteler]:
> > "User" comment: I would suggest to always activate the NULL file
compression by default, anything else would be confusing.
>
> --> not yet reflected in the patch (so, the meaning of the var
`GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
`GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
time and would like to get rid of wasting too much disk space.

That concerns #2349.
>
> Second:
>
> We need a patch for GRASS GIS 7.0 for backward compatibility, at least
for the NULL file compression. Which leads to the suggestion that either
RLE or BZIP should be used here since these two compressors are already
present in releasebranch70.

No, neither the generic RLE algorithm nor BZIP support is available in
relbr70. Note that CELL RLE compression is rasterlib-internal and tailored
for integer numbers with variable byte size. Therefore it does not make
sense to use the rasterlib-internal RLE method to compress null files
where the informative unit is one bit, not 1-4 bytes. For relbr70, null
files could be (un-)compressed with ZLIB's DEFLATE, other methods would
not be supported.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:36>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 16, 2015, 11:42am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

Replying to [comment:36 mmetz]:
> For relbr70, null files could be (un-)compressed with ZLIB's DEFLATE,
other methods would not be supported.

OK, also fine as long as one compliant solution exists.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:37>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 18, 2015, 8:43am

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by mmetz):

Replying to [comment:35 neteler]:
> Sorry for the delay.
>
> Replying to [comment:34 sprice]:
> > For "backward compatibility" do you mean files written by grass71 can
be read by grass70?
>
> Yes, exactly.
>
> > Or would you like a similar patch to also work with the grass70
branch?
>
> Perhaps in future, once we know that the grass71 implementation runs
well on all OS.

I have added different compression methods (LZ4, BZIP2) in r67212-16.
Please test.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:38>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 27, 2015, 7:18pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

(requires: r65272, r65273, r65322, r65323, r65489, r65490, r65491, r65775
from #2349)

For the record, all changesets related to this ticket which I could
identify (for a future backport):

r67212, r67213, r67214, r67215, r67216, r67217, r67218, r67254, r67295,
r67296

Note: In a local backport, I found a merge issue between r67214 and r67215
which I could not resolve. Any changeset missing from my list?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:39>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 28, 2015, 4:44pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

'''To summarize the current state'''

We currently have the following compression methods in trunk (according to
lib/gis/compress.c; set by `export GRASS_COMPRESSOR=XXX`):

  * `NONE` (uncompressed)
  * `RLE` (generic Run-Length Encoding of single bytes)
  * `ZLIB` (DEFLATE, good speed and compression)
    * with zlib compression levels (`export GRASS_ZLIB_LEVEL=X`): -1..9 (-1
is default which is level 6)
    * Notes from #comment:23: ZLIB level = 0 tells ZLIB to copy the data as
is from source to destination. With CELL maps, the rasterlib will then
still trim high zero bytes with trim_bytes() which can already reduce the
data size considerably, but ZLIB will not compress the data.
  * `LZ4` (fastest, low compression)
  * `BZIP2` (slowest, high compression)

NULL file compression: At time it must be explicitly turned on (IMHO it
should become the default) with
* `export GRASS_COMPRESS_NULLS=1`

Backward NULL file compression compatibility could be implemented in
relbranch70 only by ZLIB's DEFLATE (see #comment:36). Hence ZLIB may
qualify for the default NULL compression algorithm.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:40>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 28, 2015, 5:02pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by neteler):

Some MODIS LST benchmark, using a European LST coverage from
http://gis.cri.fmach.it/eurolst/

'''File size tests''': trunk, r67403 (build_date=2015-12-28)

{{{
# dataset
r.univar -g lst_2002_196_average
n=188341794
null_cells=226948851
cells=415290645
min=-481
max=4570
...

########################
#### RLE
export GRASS_COMPRESSOR=RLE
export GRASS_COMPRESS_NULLS=1

ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
-rw-r--r-- 1 neteler gis 10 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 51923025 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/null
-rw-r--r-- 1 neteler gis 44 Sep 22 15:15
modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
-rw-r--r-- 1 neteler gis 10 Dec 28 15:51
modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 1423931 Dec 28 15:51
modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

ls -la modis_lst_daily*/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Jun 9 2014
modis_lst_daily/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 529008029 Dec 28 15:51
modis_lst_daily_compr/cell/lst_2002_196_average

########################
#### BZIP Level 1
export GRASS_COMPRESSOR=ZLIB
export GRASS_ZLIB_LEVEL=1
export GRASS_COMPRESS_NULLS=1

ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
-rw-r--r-- 1 neteler gis 10 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 51923025 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/null
-rw-r--r-- 1 neteler gis 44 Sep 22 15:15
modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
-rw-r--r-- 1 neteler gis 10 Dec 28 16:49
modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 1579674 Dec 28 16:49
modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

ls -la modis_lst_daily*/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Jun 9 2014
modis_lst_daily/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 263256860 Dec 28 16:49
modis_lst_daily_compr/cell/lst_2002_196_average

########################
#### BZIP Level 6 (default)
export GRASS_COMPRESSOR=ZLIB
export GRASS_ZLIB_LEVEL=6
export GRASS_COMPRESS_NULLS=1

ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
-rw-r--r-- 1 neteler gis 10 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 51923025 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/null
-rw-r--r-- 1 neteler gis 44 Sep 22 15:15
modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
-rw-r--r-- 1 neteler gis 10 Dec 28 17:54
modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 1358116 Dec 28 17:54
modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

ls -la modis_lst_daily*/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Jun 9 2014
modis_lst_daily/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Dec 28 17:54
modis_lst_daily_compr/cell/lst_2002_196_average

########################
#### LZ4
export GRASS_COMPRESSOR=LZ4
export GRASS_COMPRESS_NULLS=1

ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
-rw-r--r-- 1 neteler gis 10 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 51923025 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/null
-rw-r--r-- 1 neteler gis 44 Sep 22 15:15
modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
-rw-r--r-- 1 neteler gis 10 Dec 28 17:52
modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 1573603 Dec 28 17:52
modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

ls -la modis_lst_daily*/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Jun 9 2014
modis_lst_daily/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 362932376 Dec 28 17:52
modis_lst_daily_compr/cell/lst_2002_196_average

########################
#### BZIP2
export GRASS_COMPRESSOR=BZIP2
export GRASS_COMPRESS_NULLS=1

ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
-rw-r--r-- 1 neteler gis 10 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 51923025 Jun 9 2014
modis_lst_daily/cell_misc/lst_2002_196_average/null
-rw-r--r-- 1 neteler gis 44 Sep 22 15:15
modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
-rw-r--r-- 1 neteler gis 10 Dec 28 17:49
modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
-rw-r--r-- 1 neteler gis 1824531 Dec 28 17:49
modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

ls -la modis_lst_daily*/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 257730745 Jun 9 2014
modis_lst_daily/cell/lst_2002_196_average
-rw-r--r-- 1 neteler gis 180209506 Dec 28 17:49
modis_lst_daily_compr/cell/lst_2002_196_average
}}}

Overall, ZLIB at Level 1 performs well followed by LZ4 (BZIP2 compresses
best but was very slow; ZLIB level 6's small size gain over ZLIB level 1
is "paid" by performance).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:41>
GRASS GIS <https://grass.osgeo.org>

ecodiv · December 28, 2015, 5:47pm

On 28-12-15 17:44, GRASS GIS wrote:

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
   Reporter: sprice | Owner: grass-dev@…
       Type: enhancement | Status: new
   Priority: normal | Milestone: 7.1.0
  Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
        CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

I was assuming this is for all platforms, but see above MacOSX being ticked as platform?

Comment (by neteler):

  '''To summarize the current state'''

  We currently have the following compression methods in trunk (according to
  lib/gis/compress.c; set by `export GRASS_COMPRESSOR=XXX`):

   * `NONE` (uncompressed)
   * `RLE` (generic Run-Length Encoding of single bytes)
   * `ZLIB` (DEFLATE, good speed and compression)
     * with zlib compression levels (`export GRASS_ZLIB_LEVEL=X`): -1..9 (-1
  is default which is level 6)
     * Notes from #comment:23: ZLIB level = 0 tells ZLIB to copy the data as
  is from source to destination. With CELL maps, the rasterlib will then
  still trim high zero bytes with trim_bytes() which can already reduce the
  data size considerably, but ZLIB will not compress the data.
   * `LZ4` (fastest, low compression)
   * `BZIP2` (slowest, high compression)

  NULL file compression: At time it must be explicitly turned on (IMHO it
  should become the default) with
   * `export GRASS_COMPRESS_NULLS=1`

If I want to test this, is there an overview page how to use this? E.g., do I need to run the above every time I start GRASS? And will it then compress every NULL file when opening, or is there a way to batch compress all NULL files in a mapset for example. And if compressed, can I only open the raster layers when compression is (explicitly) enabled?

  Backward NULL file compression compatibility could be implemented in
  relbranch70 only by ZLIB's DEFLATE (see #comment:36). Hence ZLIB may
  qualify for the default NULL compression algorithm.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:40>
GRASS GIS <https://grass.osgeo.org>

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

GDAL · January 12, 2016, 10:16pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by martinl):

Replying to [comment:33 neteler]:
> --> not yet reflected in the patch (so, the meaning of the var
`GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
`GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
time and would like to get rid of wasting too much disk space.

Or just perform NULL compression if environmental variable
`GRASS_COMPRESS_NULLS` is not defined or defined and set to `1`, otherwise
do not perform compression.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:42>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 18, 2016, 10:45pm

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter: sprice | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
       CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------

Comment (by mmetz):

Replying to [comment:41 neteler]:
>
> Overall, ZLIB at Level 1 performs well followed by LZ4 (BZIP2 compresses
best but was very slow; ZLIB level 6's small size gain over ZLIB level 1
is "paid" by performance).

Now we have compression options for different use cases. For large raster
maps (many columns, e.g. + 20,000), BZIP2 should provide the best
compression and should require the least disk space. If disk space
consumption and I/O speed do not matter, LZ4 could be used. The best
compression method depends on the size of the raster to be compressed, the
I/O reading and writing speed, and the available storage capacity. ZLIB
level 1 provides a pretty good compromise. Optimizing raster map
compression is now possible but requires testing with the current raster
map(s) and the available storage infrastructure (I/O speed + capacity).

TODO: documentation + r.compress

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:43>
GRASS GIS <https://grass.osgeo.org>