Thanks. It would be good to now also pave the way to compress the NULL
files by default (chose default method to be able to have read-support in
relbr70). See also wish above and here:
Replying to [comment:45 neteler]:
> Thanks. It would be good to now also pave the way to compress the NULL
files by default (chose default method to be able to have read-support in
relbr70).
Why have this also in relbr70? Backporting new features from trunk to a
release branch is a bad idea and against GRASS policy, I thought. You
could instead release 7.1 if you want these features in a release branch.
ZLIB is not really the best method for null bits compression. With large
chunks of NULL values and/or large chunks of non-NULL values, the new RLE
method seems to be as fast as no null file compression and provides the
best compression. For special cases, LZ4 should compress better than RLE
at the same speed.
Some tests with a CELL map (MODIS LST Europe) with 18711 rows and 22195
columns using r.compress (processing time and size of the null2 file):
Replying to [comment:46 mmetz]:
> Why have this also in relbr70? Backporting new features from trunk to a
release branch is a bad idea and against GRASS policy,
I wrote only that there must be one method to *read* compressed NULL files
in G70. Not much of a big backport if I recall a personal discussion we
had on this topic last year. That's all I meant above.
Or we need to release trunk as G8 in order to be allowed to break raster
readability compatibility completely.
Replying to [comment:47 neteler]:
> Replying to [comment:46 mmetz]:
> > Why have this also in relbr70? Backporting new features from trunk to
a release branch is a bad idea and against GRASS policy,
>
> I wrote only that there must be one method to *read* compressed NULL
files in G70. Not much of a big backport if I recall a personal discussion
we had on this topic last year. That's all I meant above.
Even only read support in G70 will require quite some changes to the
rasterlib.
>
> Or we need to release trunk as G8 in order to be allowed to break raster
readability compatibility completely.
In your opinion:
* Adding new compression methods in G71 for cell/fcell files that are not
readable by G70 is ok.
* Compression of null files in G71 not readable by G70 is not ok.
That does not make sense to me. Please explain.
BTW, the new compression methods for cell/fcell files available in trunk
have a much larger impact on performance and disk space requirements than
null file compression.
Replying to [comment:49 neteler]:
> Replying to [comment:48 mmetz]:
> > In your opinion:
> > * Adding new compression methods in G71 for cell/fcell files that are
not readable by G70 is ok.
>
> No. If the format is broken, we need to call it G8 and provide
converters.
If we call it G8, converters are not needed because trunk as it is now can
read G7 and G6 rasters.
Assuming trunk should be released as G7.1:
For full compatibility of relbr70 with trunk, not only read support for
compressed null files would need to be backported, but also and more
importantly read support for compressed cell/fcell files. These are new
features at library level involving quite a few changes.
The current default settings in trunk are compatible with relbr70, and I
do not see a reason to change the respective default settings in trunk.
What exactly is the reason why null file compression should be enabled by
default in trunk?
I will not backport the changes I did to trunk with regard to cell/cell
file and null file compression because these are new features which should
IMHO not go into an existing release branch. Someone else would need to do
the backporting.
IMHO, the new compression methods for cell/fcell files are more important
than null file compression because the potential gains in disk space and
processing speed are much higher. Recompressing a large cell file from
ZLIB to BZIP2 can gain disk space several times larger than the
corresponding uncompressed(!) null file. In relation, a compressed null
file would only be a small added bonus but not a substantial gain in free
disk space. On the other hand, compressing cell/fcell files with LZ4 can
substantially speed up processing time.
Short answer: release trunk now as G8, no need for converters because
backward compatibility exists.
Replying to [comment:50 mmetz]:
> Replying to [comment:49 neteler]:
> > Replying to [comment:48 mmetz]:
> > > In your opinion:
> > > * Adding new compression methods in G71 for cell/fcell files that
are not readable by G70 is ok.
> >
> > No. If the format is broken, we need to call it G8 and provide
converters.
>
> If we call it G8, converters are not needed because trunk as it is now
can read G7 and G6 rasters.
>
> Assuming trunk should be released as G7.1:
>
> For full compatibility of relbr70 with trunk, not only read support for
compressed null files would need to be backported, but also and more
importantly read support for compressed cell/fcell files. These are new
features at library level involving quite a few changes.
-1 to backporting. Only bug fixes should go into relbr70.
>
> The current default settings in trunk are compatible with relbr70, and I
do not see a reason to change the respective default settings in trunk.
+1, as long as the default settings lead to compatible maps, then I do not
think any backporting is needed.
>What exactly is the reason why null file compression should be enabled by
default in >trunk?
Again +1. I don't understand all the effects and issues involved, but null
file compression seems to be something that is mostly needed for specific
big data situations. People dealing with such situations can alter
specific settings.
>
> I will not backport the changes I did to trunk with regard to cell/cell
file and null file compression because these are new features which should
IMHO not go into an existing release branch.
Replying to [comment:57 wenzeslaus]:
> This seems like a great success!
>
> Seth, do you want to add another method?
>
Other possible methods are LZ4H and lzma from xz, but IMHO the currently
implemented methods are sufficient.
> MarkusM, is there something else remaining?
I don't think so.
>
> MarkusN, anything you want to change or document?
Maybe some documentation that BZIP2 is not always providing the highest
compression ratio, it needs to be evaluated for each dataset separately.
>
> If everybody is OK, we can close this.
Replying to [comment:58 mmetz]:
> Maybe some documentation that BZIP2 is not always providing the highest
compression ratio, it needs to be evaluated for each dataset separately.
Replying to [comment:59 neteler]:
> Replying to [comment:58 mmetz]:
> > Maybe some documentation that BZIP2 is not always providing the
highest compression ratio, it needs to be evaluated for each dataset
separately.
>
> Probably the notes on BZIP2 are sufficient here?
>
> https://grass.osgeo.org/grass72/manuals/r.compress.html#compression-
algorithm-details
Replying to [comment:61 neteler]:
> Replying to [comment:60 mmetz]:
> > Notes on NULL file compression could be added to the raster intro.
>
> Done in r70669 and r70670.
>
> Proposal: I would suggest to make NULL file compression the default in
trunk.
Suggested tests:
* have a raster map with NULL cells without NULL compression
export GRAS_COMPRESS_NULLS=1
test the result of r.compress
test the result of r.null -z
* have a raster map with NULL cells with NULL compression
unset GRAS_COMPRESS_NULLS
test the result of r.compress
test the result of r.null -z
* have two raster maps, one with compressed NULL cells, one with
uncompressed NULL cells