Hi,
I'd like to turn off compression for all maps. Is this possible with g.gisenv or environment variables? The g.gisenv manual only shows how to choose among compression methods, not disable it.
This does not seem to work:
g.gisenv set="GRASS_COMPRESSOR=none" # optimize for speed
I can decompress one map with:
r.compress -u map=foo
But I'm trying to skip the time spent compressing and decompressing altogether. I could use a small speed boost, and this seems like an easy way to speed things up - I can afford the disk space for all the temporary intermediate maps.
Thanks,
-k.
Hm. Answer: No
I just did:
export GRASS_COMPRESSOR=none
And then get this error message:
WARNING: No compression is not supported for GRASS raster maps, using default ZLIB
Seems like a shame to not have this as an option. I'll try speeding it up by tweaking the compression method and amount.
-k.
On 2020-02-29 at 08:37 -08, Ken Mankoff <mankoff@gmail.com> wrote...
Hi,
I'd like to turn off compression for all maps. Is this possible with
g.gisenv or environment variables? The g.gisenv manual only shows how
to choose among compression methods, not disable it.
This does not seem to work:
g.gisenv set="GRASS_COMPRESSOR=none" # optimize for speed
I can decompress one map with:
r.compress -u map=foo
But I'm trying to skip the time spent compressing and decompressing
altogether. I could use a small speed boost, and this seems like an
easy way to speed things up - I can afford the disk space for all the
temporary intermediate maps.
Thanks,
-k.
Hi Ken,
That sounds like a good experiment and I think there would be general interest in what you find. However, please note that less compression may not mean faster, because then there is more data to write and read and I/O is generally a significant bottleneck. Markus Metz may say more about this as he did tests when he was implementing the new compressions, but you might be able to find some posts on this on the grass-dev mailing list.
Best,
Vaclav
On Sat, Feb 29, 2020 at 11:40 AM Ken Mankoff <mankoff@gmail.com> wrote:
Hm. Answer: No
I just did:
export GRASS_COMPRESSOR=none
And then get this error message:
WARNING: No compression is not supported for GRASS raster maps, using default ZLIB
Seems like a shame to not have this as an option. I’ll try speeding it up by tweaking the compression method and amount.
-k.
On 2020-02-29 at 08:37 -08, Ken Mankoff <mankoff@gmail.com> wrote…
Hi,
I’d like to turn off compression for all maps. Is this possible with
g.gisenv or environment variables? The g.gisenv manual only shows how
to choose among compression methods, not disable it.
This does not seem to work:
g.gisenv set=“GRASS_COMPRESSOR=none” # optimize for speed
I can decompress one map with:
r.compress -u map=foo
But I’m trying to skip the time spent compressing and decompressing
altogether. I could use a small speed boost, and this seems like an
easy way to speed things up - I can afford the disk space for all the
temporary intermediate maps.
Thanks,
-k.
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
Hi Ken,
turning off compression for all newly generated maps is indeed not supported. If you don’t care about disk usage and disk speed but processing time, LZ4 is the compression method of choice. LZ4 compresses in near-real time, i. e. in GRASS GIS there is no measurable CPU speed penalty, only increased disk IO because more data need to be written and subsequently read. I am using LZ4 myself as raster compression in workflows for temporary intermediate maps and change the raster compression method before writing out final results.
Currently, the default GRASS raster compression method can only be changed with the environment variable GRASS_COMPRESSOR.
Markus M
On Sun, Mar 1, 2020 at 4:21 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
Hi Ken,
That sounds like a good experiment and I think there would be general interest in what you find. However, please note that less compression may not mean faster, because then there is more data to write and read and I/O is generally a significant bottleneck. Markus Metz may say more about this as he did tests when he was implementing the new compressions, but you might be able to find some posts on this on the grass-dev mailing list.
Best,
Vaclav
On Sat, Feb 29, 2020 at 11:40 AM Ken Mankoff <mankoff@gmail.com> wrote:
Hm. Answer: No
I just did:
export GRASS_COMPRESSOR=none
And then get this error message:
WARNING: No compression is not supported for GRASS raster maps, using default ZLIB
Seems like a shame to not have this as an option. I’ll try speeding it up by tweaking the compression method and amount.
-k.
On 2020-02-29 at 08:37 -08, Ken Mankoff <mankoff@gmail.com> wrote…
Hi,
I’d like to turn off compression for all maps. Is this possible with
g.gisenv or environment variables? The g.gisenv manual only shows how
to choose among compression methods, not disable it.
This does not seem to work:
g.gisenv set=“GRASS_COMPRESSOR=none” # optimize for speed
I can decompress one map with:
r.compress -u map=foo
But I’m trying to skip the time spent compressing and decompressing
altogether. I could use a small speed boost, and this seems like an
easy way to speed things up - I can afford the disk space for all the
temporary intermediate maps.
Thanks,
-k.
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user
Hi,
Thanks for the replies. Yes I found LZ4 to be the best. In answer to Vaclav suggesting general interest, here are some code and results so you can test this on your own system which will have different disk/CPU/RAM speeds. I am only reporting time, not disk usage here.
I'm not totally sure how to find the size of a GRASS raster from within GRASS. I could go to the OS to find it. A better report would include a scatter plot of time v. size, but I only cared about time.
Summary: Use LZ4 compression within GRASS. For any maps that already exist, it isn't clear if LZ4 or uncompressed is faster. They seem about equal on my system using this as a test:
r.mapcalc "compressed = 42"
r.mapcalc "uncompressed = 42"
r.compress -u map=uncompressed
/usr/bin/time -f %E parallel -j 1 -N 0 r.mapcalc '"foo = compressed * 2"' ::: $(seq 10)
/usr/bin/time -f %E parallel -j 1 -N 0 r.mapcalc '"foo = uncompressed * 2"' ::: $(seq 10)
-k.
grass -c EPSG:3413 ./G
g.region w=0 e=10000 s=0 n=10000 res=1 -pa
export GRASS_OVERWRITE=1
for C in RLE LZ4 BZIP2; do
export GRASS_COMPRESSOR=${C}
echo -n "${C} ":
/usr/bin/time -f %E parallel -j 1 -N 0 r.mapcalc '"foo = 42"' ::: $(seq 10)
done
export GRASS_COMPRESSOR=ZLIB
for ZL in $(seq -1 10); do
export GRASS_ZLIB_LEVEL=${ZL}
echo -n "ZLIB ${ZL} ":
/usr/bin/time -f %E parallel -j 1 -N 0 r.mapcalc '"foo = 42"' ::: $(seq 10)
done
RLE :0:17.39
LZ4 :0:14.51
BZIP2 :0:24.05
ZLIB -1 :0:22.49
ZLIB 0 :0:19.14
ZLIB 1 :0:20.31
ZLIB 2 :0:20.05
ZLIB 3 :0:19.87
ZLIB 4 :0:22.72
ZLIB 5 :0:22.49
ZLIB 6 :0:22.25
ZLIB 7 :0:22.32
ZLIB 8 :0:22.26
ZLIB 9 :0:22.67
ZLIB 10 :0:19.77
On Mon, Mar 2, 2020 at 4:58 PM Ken Mankoff <mankoff@gmail.com> wrote:
I’m not totally sure how to find the size of a GRASS raster from within GRASS.
There is no tool in GRASS that reports the disk usage of a GRASS raster map. r.compress only reports the change in disk usage.
I could go to the OS to find it.
That’s what I do too to find out disk usage of a GRASS raster map.
A better report would include a scatter plot of time v. size, but I only cared about time.
Summary: Use LZ4 compression within GRASS.
Only if you don’t care about disk space and if IO speed does not matter. Particularly when sharing or archiving data, other compression methods reducing disk space are more suitable.
Markus M
On Mon, Mar 2, 2020 at 9:35 PM Markus Metz
<markus.metz.giswork@gmail.com> wrote:
On Mon, Mar 2, 2020 at 4:58 PM Ken Mankoff <mankoff@gmail.com> wrote:
>
> I'm not totally sure how to find the size of a GRASS raster from within GRASS.
There is no tool in GRASS that reports the disk usage of a GRASS raster map. r.compress only reports the change in disk usage.
(there is an enhancement wish ticket for that:
https://trac.osgeo.org/grass/ticket/3917
)
markusN