#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
Reporter: sprice | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------
Comment (by mmetz):
Replying to [comment:20 wenzeslaus]:
> Replying to [comment:19 mmetz]:
> > I have implemented something like this recently and added support for
LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
> >
> > I am attaching a patch for trunk r66775 and an archive with new files
to go to lib/gis
>
> The design in the patch looks really good. I did tests and benchmark but
it was not as successful as I hoped for.
>
> The benchmark was the same as [comment:10 before] but modified for this
patch. It is more for testing than benchmark anyway. It was on 30,000,000
cells but perhaps the previous one was on more and it is not completely
precise overall due to some other computations running at the same time
(although the result is from 10 runs aggregated by ''perf'').
>
> || type || write || read ||
> || NONE || 2.58 || 0.72 ||
> || ZLIB || 1.52 || 0.93 ||
> || LZ4 || 1.56 || 0.85 ||
>
Some more explanation about the proposed new mechanism:
The proposed `G_compress()` interface provides a generic mechanism to data
compression in libgis, not restricted to raster data but generic. Built-in
compression methods would be no compression, RLE, ZLIB, ZL4. BZIP2
compression would be available if GRASS is configured --with-bzlib. Other
compression methods could be added by cloning lib/gis/flate.c and adding
new `G_*_compress()` and `G_*_expand()` functions to
lib/gis/compress.[h|c]. The raster lib does not need to be modified any
more.
As before, the rasterlib makes only partial use of the generic compression
methods: no compression and RLE is handled by the rasterlib internally,
and RLE is not supported for fp maps. Creating uncompressed raster maps
has been and should be only possible with `Rast_open_new_uncompressed()`.
That means that the behaviour of the rasterlib with using
GRASS_COMPRESSOR=NONE needs to be defined: really use no compression or
use default compression instead? Using GRASS_COMPRESSOR=RLE affects only
new CELL maps. For fp maps, compression_type = 1 (RLE) is as before
interpreted as ZLIB compression.
If you want to know the original amount of data passed to any compressor,
you need to use
{{{
GRASS_COMPRESSOR=ZLIB
GRASS_ZLIB_LEVEL=0
}}}
ZLIB level = 0 tells ZLIB to copy the data as is from source to
destination. With CELL maps, the rasterlib will then still trim high zero
bytes with trim_bytes() which can already reduce the data size
considerably, but ZLIB will not compress the data.
I modified gislib_compressor_benchmark.sh to use
{{{
GRASS_COMPRESSOR=ZLIB
GRASS_ZLIB_LEVEL=0
}}}
for no compression and discarded RLE because it is inefficient for CELL
maps and not supported for fp maps. I tested also ZLIB levels 1 (fastest)
and 6 (ZLIB default).
I used the nc_basic_spm_grass7 location and set the region with
{{{
g.region -p rast=elevation res=2.5
}}}
resulting in 32,000,000 cells
The test raster was generated with
{{{
r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
seed=100
}}}
random numbers very difficult to compress.
The write and read columns in the tables below have seconds as unit.
|| compressor || size MB || size % || write || read ||
|| NONE || 259.2 || 100 || 5.2 || 1.4 ||
|| ZLIB 1 || 247.9 || 95.6 || 14.3 || 2.5 ||
|| ZLIB 6 || 246.9 || 95.3 || 16.3 || 2.4 ||
|| LZ4 || 259.2 || 100 || 4.5 || 1.1 ||
|| BZIP2 || 249.4 || 96.2 || 63.4 || 19.9 ||
LZ4 is the fastest method, no method is really the best because these
random numbers could not be compressed to less than 95% of the original
size.
The next test raster was generated with
{{{
r.mapcalc expression="test_rast2_z_base = elevation"
}}}
which had 4x4 blocks of identical raster values, should be easy to
compress
|| compressor || size MB || size % || write || read ||
|| NONE || 259.2 || 100 || 4.2 || 1.1 ||
|| ZLIB 1 || 41.8 || 16.1 || 4.7 || 1.6 ||
|| ZLIB 6 || 32.1 || 12.4 || 11.1 || 1.4 ||
|| LZ4 || 71.5 || 27.6 || 2.2 || 0.9 ||
|| BZIP2 || 49.8 || 19.2 || 28.0 || 7.1 ||
LZ4 was again the fastest, and the best was ZLIB level 6. Here, the
performance of BZIP2 was not convincing: by far the slowest and not as
good as ZLIB.
Then I tested with MODIS land surface temperature for Europe, a bit more
than 400,000,000 cells:
LST as CELL
|| compressor || size MB || size % || write || read ||
|| NONE || 829 || 100 || 28.5 || 14.8 ||
|| ZLIB 1 || 269 || 32.4 || 30.5 || 14.9 ||
|| ZLIB 6 || 261 || 31.5 || 36.7 || 16.5 ||
|| LZ4 || 366 || 44.1 || 20.0 || 13.0 ||
|| BZIP2 || 175 || 21.1 || 89.8 || 29.2 ||
LZ4 was the fastest, BZIP2 was the best.
LST as DCELL
|| compressor || size MB || size % || write || read ||
|| NONE || 3300 || 100 || 85.5 || 52.9 ||
|| ZLIB 1 || 503 || 15.2 || 62.7 || 23.9 ||
|| ZLIB 6 || 370 || 11.2 || 129.8 || 21.6 ||
|| LZ4 || 629 || 19.1 || 29.5 || 13.9 ||
|| BZIP2 || 196 || 5.9 || 221 || 51.1 ||
Again, LZ4 was the fastest, BZIP2 was the best.
I am interested in having BZIP2 because for these LST data it compresses
30 - 50% better than the second best (ZLIB level 6).
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:23>
GRASS GIS <https://grass.osgeo.org>