[GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

mlennert · December 4, 2019, 9:28am

Hi Markus,

In recent days, I've been confronted several times with the issue of people trying to share data among themselves, but using different versions of GRASS, and so raster data compressed in a more recent version of GRASS was not usable in an older version of GRASS.

Now, I agree that generally the solution is to tell people to use the latest and greatest, but this is not always possible / it is not necessarily highest on the list of priorities of people to see how they can install the latest version of GRASS within their particular environment.

Obviously, those with the latest version of GRASS can simple recompress using ZLIB. However, compression method is defined as an environment variable. This is somewhat daunting to many MS Windows users out there. Is there any specific reason that lead to the choice of not using a parameter to allow the choice of compression method (possibly to override a default that is still defined by an environment variable) ?

Moritz

marisn · December 5, 2019, 6:30am

What should be the solution? Moving to choice set by g.gisenv?

Māris.

trešd., 2019. g. 4. dec., plkst. 11:28 — lietotājs Moritz Lennert
(<mlennert@club.worldonline.be>) rakstīja:

Hi Markus,

In recent days, I've been confronted several times with the issue of
people trying to share data among themselves, but using different
versions of GRASS, and so raster data compressed in a more recent
version of GRASS was not usable in an older version of GRASS.

Now, I agree that generally the solution is to tell people to use the
latest and greatest, but this is not always possible / it is not
necessarily highest on the list of priorities of people to see how they
can install the latest version of GRASS within their particular environment.

Obviously, those with the latest version of GRASS can simple recompress
using ZLIB. However, compression method is defined as an environment
variable. This is somewhat daunting to many MS Windows users out there.
Is there any specific reason that lead to the choice of not using a
parameter to allow the choice of compression method (possibly to
override a default that is still defined by an environment variable) ?

Moritz

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

mlennert · December 5, 2019, 6:59am

Le Thu, 5 Dec 2019 08:30:43 +0200,
Maris Nartiss <maris.gis@gmail.com> a écrit :

What should be the solution? Moving to choice set by g.gisenv?

As mentioned in my mail, I suggest to add a parameter to the module
which allows to override the default setting when compressing a
specific raster map.

Using a GRASS environment variable instead of a shell variable might be
an idea to make setting the default easier.

Moritz

trešd., 2019. g. 4. dec., plkst. 11:28 — lietotājs Moritz Lennert
(<mlennert@club.worldonline.be>) rakstīja:
>
> Hi Markus,
>
> In recent days, I've been confronted several times with the issue of
> people trying to share data among themselves, but using different
> versions of GRASS, and so raster data compressed in a more recent
> version of GRASS was not usable in an older version of GRASS.
>
> Now, I agree that generally the solution is to tell people to use
> the latest and greatest, but this is not always possible / it is not
> necessarily highest on the list of priorities of people to see how
> they can install the latest version of GRASS within their
> particular environment.
>
> Obviously, those with the latest version of GRASS can simple
> recompress using ZLIB. However, compression method is defined as an
> environment variable. This is somewhat daunting to many MS Windows
> users out there. Is there any specific reason that lead to the
> choice of not using a parameter to allow the choice of compression
> method (possibly to override a default that is still defined by an
> environment variable) ?
>
> Moritz
>
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Département Géosciences, Environnement et Société Université Libre de
Bruxelles Bureau: S.DB.6.138
CP 130/03
Av. F.D. Roosevelt 50
1050 Bruxelles
Belgique

tél. + 32 2 650.68.12 / 68.11 (secr.)
fax + 32 2 650.68.30

Markus_Metz · December 9, 2019, 9:50pm

Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert <mlennert@club.worldonline.be> wrote:

Hi Markus,

In recent days, I’ve been confronted several times with the issue of
people trying to share data among themselves, but using different
versions of GRASS, and so raster data compressed in a more recent
version of GRASS was not usable in an older version of GRASS.

Now, I agree that generally the solution is to tell people to use the
latest and greatest, but this is not always possible / it is not
necessarily highest on the list of priorities of people to see how they
can install the latest version of GRASS within their particular environment.

Obviously, those with the latest version of GRASS can simple recompress
using ZLIB. However, compression method is defined as an environment
variable. This is somewhat daunting to many MS Windows users out there.
Is there any specific reason that lead to the choice of not using a
parameter to allow the choice of compression method (possibly to
override a default that is still defined by an environment variable) ?

Such a parameter would need to be added to every module creating raster output.

The reason for an environment variable is historical: previously, CELL maps were by default compressed with RLE, this could be changed with the env var GRASS_INT_ZLIB. This mechanism of using an env var for raster compression has been extended.

Other solutions than an env var would be to use g.gisenv/$GISRC to set the default compression method per user or a mapset-specific compression method (to be set with a modified r.compress?) stored in the mapset’s VAR file.

Regarding MS Windows users + raster compression: I think it is reasonable to assume at least GRASS 7.6 which has support all compression methods available in GRASS.

Markus M

mlennert · December 10, 2019, 5:47am

On 9/12/19 22:50, Markus Metz wrote:

Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert <mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be>> wrote:
>
> Hi Markus,
>
> In recent days, I've been confronted several times with the issue of
> people trying to share data among themselves, but using different
> versions of GRASS, and so raster data compressed in a more recent
> version of GRASS was not usable in an older version of GRASS.
>
> Now, I agree that generally the solution is to tell people to use the
> latest and greatest, but this is not always possible / it is not
> necessarily highest on the list of priorities of people to see how they
> can install the latest version of GRASS within their particular environment.
>
> Obviously, those with the latest version of GRASS can simple recompress
> using ZLIB. However, compression method is defined as an environment
> variable. This is somewhat daunting to many MS Windows users out there.
> Is there any specific reason that lead to the choice of not using a
> parameter to allow the choice of compression method (possibly to
> override a default that is still defined by an environment variable) ?

Such a parameter would need to be added to every module creating raster output.

My request is more linked to use cases where one would like to share data (e.g. with r.pack) with other GRASS GIS users who do not necessarily have access to the same compression method, not necessarily to changing the default compression method. I was just wondering whether it might be easily possible to just implement r.compress method= as a quick way to recompress a specific map with a chosen method, overriding the default method. Currently, to do that, you have to change the default method by changing the env variable, run r.compress, then change the variable back to the value one wishes generally to use as default.

Obviously, you can always just export as tiff and share that, but that just feels less elegant. Anyway, this is probably somewhat of a luxury problem

Moritz

Anika_Bettge · December 10, 2019, 10:37am

Hello all,

I try to import with GRASS78 in parallel Sentinel-2 data with "i.sentinel.import -r" (Multicore VM (32 cores -> 31 parallel processes)).

Sometimes the following error occurs (note the formatting comes from deploying in actinia):
          ...
          "99..Reprojecting <T32ULB_20181010T104019_B08_10m>...",
          "96..100",
          "90..99..96..100",
          "99..93..100",
          "99..100",
          "96..96..99..100",
          "99..100",
          "Estimated target resolution for input band <T32ULC_20180702T104019_B04_10m>: 9.998178838141047",
          "Estimated target resolution for input band <T32ULC_20181010T104019_B04_10m>: 9.998178838141047",
          "Estimated target resolution for input band <T32ULB_20180930T104019_B08_10m>: 9.998178838136036",
          "Estimated target resolution for input band <T32ULC_20180806T104021_B04_10m>: 9.998178838141047",
          "Estimated target resolution for input band <T31UGT_20180806T104021_B08_10m>: 9.26372380446332",
          "Estimated target resolution for input band <T31UGT_20181010T104019_B08_10m>: 9.26372380446332",
          "ERROR: Region file mapset_a6335f7028114ee9989530cb55874785//WIND is empty", <---- here
          "ERROR: Region file mapset_a6335f7028114ee9989530cb55874785//WIND is empty",
          "Using given resolution for input band <T32ULC_20180702T104019_B04_10m>: 10.0",
          "Using given resolution for input band <T32ULC_20181010T104019_B04_10m>: 10.0",
          "Using given resolution for input band <T32ULC_20180806T104021_B04_10m>: 10.0",
          "Reprojecting <T32ULC_20181010T104019_B04_10m>...",
          "Reprojecting <T32ULC_20180702T104019_B04_10m>...",
          "Traceback (most recent call last):",
          " File \"/usr/local/grass7/scripts/r.import\", line 403, in <module>",
          " sys.exit(main())",
          " File \"/usr/local/grass7/scripts/r.import\", line 355, in main",
          " grass.use_temp_region()",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 1216, in use_temp_region",
          " run_command(\"g.region\", save=name, overwrite=True)",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 441, in run_command",
          " return handle_errors(returncode, returncode, args, kwargs)",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 343, in handle_errors",
          " returncode=returncode)",
          "grass.exceptions.CalledModuleError: Module run None g.region --o save=tmp.r.import.19907 ended with error",
          "Process ended with non-zero return code 1. See errors in the (error) output.",
          "Traceback (most recent call last):",
          " File \"/usr/local/grass7/scripts/r.import\", line 403, in <module>",
          " sys.exit(main())",
          " File \"/usr/local/grass7/scripts/r.import\", line 355, in main",
          " grass.use_temp_region()",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 1216, in use_temp_region",
          " run_command(\"g.region\", save=name, overwrite=True)",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 441, in run_command",
          " return handle_errors(returncode, returncode, args, kwargs)",
          " File \"/usr/local/grass7/etc/python/grass/script/core.py\", line 343, in handle_errors",
          " returncode=returncode)",
          "grass.exceptions.CalledModuleError: Module run None g.region --o save=tmp.r.import.20767 ended with error",
          "Process ended with non-zero return code 1. See errors in the (error) output.",
          "Processing <T32ULB_20180930T104019_B04_10m>...",
          "Using given resolution for input band <T31UGT_20181010T104019_B08_10m>: 10.0",
          "Processing <T31UGT_20180806T104021_B04_10m>...",
          "Reprojecting <T32ULC_20180806T104021_B04_10m>...",
          "Reprojecting <T31UGT_20181010T104019_B08_10m>...",
          "WARNING: Projection of dataset does not appear to match current location.",
          ...

I found out that r.import uses temp_region and that this sets an environment variable WIND_OVERRIDE:
https://github.com/OSGeo/grass/blob/4dc36bb86fa9181259fa7f9b9472f8d3bc4e787e/lib/python/script/core.py#L1281
os.environ['WIND_OVERRIDE'] = name

And in parallel processing the variable is set by each process and the temp_region can be deleted to early.

Does anyone know if this is the reason of my error and how to fix this?

Best,
Anika

Huidae_Cho · December 10, 2019, 4:06pm

One workaround may be GRASS_COMPRESSOR=method r.compress, but it won’t directly work on MS-Windows without a batch file, I think.

Huidae

···

Huidae Cho, Ph.D., GISP, PE (MD), CFM, M.ASCE
Open Source GIS Developer, GRASS GIS Development Team

Markus_Metz · December 15, 2019, 8:41pm

On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert <mlennert@club.worldonline.be> wrote:

On 9/12/19 22:50, Markus Metz wrote:

Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
<mlennert@club.worldonline.be mailto:[mlennert@club.worldonline.be](mailto:mlennert@club.worldonline.be)> wrote:

Hi Markus,

In recent days, I’ve been confronted several times with the issue of
people trying to share data among themselves, but using different
versions of GRASS, and so raster data compressed in a more recent
version of GRASS was not usable in an older version of GRASS.

Now, I agree that generally the solution is to tell people to use the
latest and greatest, but this is not always possible / it is not
necessarily highest on the list of priorities of people to see how they
can install the latest version of GRASS within their particular
environment.

Obviously, those with the latest version of GRASS can simple recompress
using ZLIB. However, compression method is defined as an environment
variable. This is somewhat daunting to many MS Windows users out there.
Is there any specific reason that lead to the choice of not using a
parameter to allow the choice of compression method (possibly to
override a default that is still defined by an environment variable) ?

Such a parameter would need to be added to every module creating raster
output.

My request is more linked to use cases where one would like to share
data (e.g. with r.pack) with other GRASS GIS users who do not
necessarily have access to the same compression method, not necessarily
to changing the default compression method. I was just wondering whether
it might be easily possible to just implement r.compress method= as a
quick way to recompress a specific map with a chosen method, overriding
the default method. Currently, to do that, you have to change the
default method by changing the env variable, run r.compress, then change

the variable back to the value one wishes generally to use as default.

recompressing an existing raster map seems like a waste of time. Why not compress it in the first place with the appropriate method?

Recompressing an existing map with a chosen method via r.compress method=XY means that a new raster map should not be compressed with the default method but another supported method. This would require a new function in lib/raster like Rast_open_new_compression_method() that overrides the default compression method. Of course this is possible, but it requires substantial changes to lib/raster.

About the use cases:
People want to share a mapset: in this case all GRASS raster maps should be compressed with the most compatible compression method (ZLIB). Currently, this can be done with an env var. Alternatively, a new mechanism via g.gisenv could be established.
Another option could be to set the default compression method per mapset, to be stored in in the mapset’s VAR file, to be set with e.g. r.compress method=.

The point is that it is fairly easy to set a suitable compression method before the raster lib is initialized, and there are different possible mechanisms apart from the current method using an env var that are easy to implement.

Obviously, you can always just export as tiff and share that, but that
just feels less elegant. Anyway, this is probably somewhat of a luxury
problem

This only a luxury problem if you don’t need to care about processing speed and disk usage. The different compression methods by now available in GRASS allow for optimization for speed (LZ4) or for optimization for disk usage (BZIP2 for larger CELL maps). An easy (easier) way to set the compression method seems reasonable.

Markus M

Moritz

mlennert · January 16, 2020, 9:56am

Coming back to this to continue the discussion:

On 15/12/19 21:41, Markus Metz wrote:

On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert <mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be>> wrote:
>
> On 9/12/19 22:50, Markus Metz wrote:
> > Hi Moritz,
> >
> > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
> > <mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be> <mailto:mlennert@club.worldonline.be>> wrote:
> > >
> > > Hi Markus,
> > >
> > > In recent days, I've been confronted several times with the issue of
> > > people trying to share data among themselves, but using different
> > > versions of GRASS, and so raster data compressed in a more recent
> > > version of GRASS was not usable in an older version of GRASS.
> > >
> > > Now, I agree that generally the solution is to tell people to use the
> > > latest and greatest, but this is not always possible / it is not
> > > necessarily highest on the list of priorities of people to see how they
> > > can install the latest version of GRASS within their particular
> > environment.
> > >
> > > Obviously, those with the latest version of GRASS can simple recompress
> > > using ZLIB. However, compression method is defined as an environment
> > > variable. This is somewhat daunting to many MS Windows users out there.
> > > Is there any specific reason that lead to the choice of not using a
> > > parameter to allow the choice of compression method (possibly to
> > > override a default that is still defined by an environment variable) ?
> >
> > Such a parameter would need to be added to every module creating raster
> > output.
>
> My request is more linked to use cases where one would like to share
> data (e.g. with r.pack) with other GRASS GIS users who do not
> necessarily have access to the same compression method, not necessarily
> to changing the default compression method. I was just wondering whether
> it might be easily possible to just implement r.compress method= as a
> quick way to recompress a specific map with a chosen method, overriding
> the default method. Currently, to do that, you have to change the
> default method by changing the env variable, run r.compress, then change
> the variable back to the value one wishes generally to use as default.

recompressing an existing raster map seems like a waste of time. Why not compress it in the first place with the appropriate method?

Because what you consider the appropriate method at the time creation of the data (and still the appropriate method for your actual work) might not be the appropriate method for sharing with someone using a different system.

Recompressing an existing map with a chosen method via r.compress method=XY means that a new raster map should not be compressed with the default method but another supported method. This would require a new function in lib/raster like Rast_open_new_compression_method() that overrides the default compression method. Of course this is possible, but it requires substantial changes to lib/raster.

Ok, I understand that this is complicated.

About the use cases:
People want to share a mapset: in this case all GRASS raster maps should be compressed with the most compatible compression method (ZLIB). Currently, this can be done with an env var. Alternatively, a new mechanism via g.gisenv could be established.

This would already make life easier for Windows users who (for the majority) are not used to env variables (and this is becoming more and more true of GNU/Linux users as well...).

Another option could be to set the default compression method per mapset, to be stored in in the mapset's VAR file, to be set with e.g. r.compress method=.

That would probably be a nice solution. One could then create an "Export" mapset which can then be easily shared.

> Obviously, you can always just export as tiff and share that, but that
> just feels less elegant. Anyway, this is probably somewhat of a luxury
> problem

This only a luxury problem if you don't need to care about processing speed and disk usage. The different compression methods by now available in GRASS allow for optimization for speed (LZ4) or for optimization for disk usage (BZIP2 for larger CELL maps). An easy (easier) way to set the compression method seems reasonable.

I didn't mean that compression is a luxury problem, but rather that since the option exists of just exporting the map as a tiff, readable by most systems (and you can chose a sharable tiff compression method at export time), all the above is probably a bit less of a necessity, and I'm just making noise

Moritz

Markus_Metz · January 16, 2020, 8:29pm

On Thu, Jan 16, 2020 at 10:56 AM Moritz Lennert <mlennert@club.worldonline.be> wrote:

Coming back to this to continue the discussion:

On 15/12/19 21:41, Markus Metz wrote:

On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert
<mlennert@club.worldonline.be mailto:[mlennert@club.worldonline.be](mailto:mlennert@club.worldonline.be)> wrote:

On 9/12/19 22:50, Markus Metz wrote:

Hi Moritz,

On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
<mlennert@club.worldonline.be mailto:[mlennert@club.worldonline.be](mailto:mlennert@club.worldonline.be)
<mailto:mlennert@club.worldonline.be
mailto:[mlennert@club.worldonline.be](mailto:mlennert@club.worldonline.be)>> wrote:

Hi Markus,

In recent days, I’ve been confronted several times with the issue of
people trying to share data among themselves, but using different
versions of GRASS, and so raster data compressed in a more recent
version of GRASS was not usable in an older version of GRASS.

Now, I agree that generally the solution is to tell people to
use the
latest and greatest, but this is not always possible / it is not
necessarily highest on the list of priorities of people to see
how they
can install the latest version of GRASS within their particular
environment.

Obviously, those with the latest version of GRASS can simple
recompress
using ZLIB. However, compression method is defined as an environment
variable. This is somewhat daunting to many MS Windows users out
there.
Is there any specific reason that lead to the choice of not using a
parameter to allow the choice of compression method (possibly to
override a default that is still defined by an environment
variable) ?

Such a parameter would need to be added to every module creating raster
output.

My request is more linked to use cases where one would like to share
data (e.g. with r.pack) with other GRASS GIS users who do not
necessarily have access to the same compression method, not necessarily
to changing the default compression method. I was just wondering whether
it might be easily possible to just implement r.compress method= as a
quick way to recompress a specific map with a chosen method, overriding
the default method. Currently, to do that, you have to change the
default method by changing the env variable, run r.compress, then change
the variable back to the value one wishes generally to use as default.

recompressing an existing raster map seems like a waste of time. Why not
compress it in the first place with the appropriate method?

Because what you consider the appropriate method at the time creation of
the data (and still the appropriate method for your actual work) might
not be the appropriate method for sharing with someone using a different
system.

True, for sharing the highest (smallest size) compression method is usually preferred, while for actual work the fastest compression method is usually preferred.

About the use cases:
People want to share a mapset: in this case all GRASS raster maps should
be compressed with the most compatible compression method (ZLIB).
Currently, this can be done with an env var. Alternatively, a new
mechanism via g.gisenv could be established.

This would already make life easier for Windows users who (for the
majority) are not used to env variables (and this is becoming more and
more true of GNU/Linux users as well…).

Another option could be to set the default compression method per
mapset, to be stored in in the mapset’s VAR file, to be set with e.g.
r.compress method=.

That would probably be a nice solution. One could then create an

“Export” mapset which can then be easily shared.

You could do the same by creating some “Export” mapset, then setting the compression method and creating raster maps to be exported in this “Export” mapset.

Implementation of a compression method set with g.gisenv is relatively easy.

Implementation of a compression method defined in a mapset’s VAR file is a bit more work and requires a mechanism to set the compression method in a mapset’s VAR file, e.g. r.compress method=.

But, if e.g. the aim is to reduce file size, the best compression method depends on the datatype and size of the individual raster map to be compressed. There is no “one size fits all” method.

Obviously, you can always just export as tiff and share that, but that
just feels less elegant. Anyway, this is probably somewhat of a luxury
problem

This only a luxury problem if you don’t need to care about processing
speed and disk usage. The different compression methods by now available
in GRASS allow for optimization for speed (LZ4) or for optimization for
disk usage (BZIP2 for larger CELL maps). An easy (easier) way to set the
compression method seems reasonable.

I didn’t mean that compression is a luxury problem, but rather that
since the option exists of just exporting the map as a tiff, readable by
most systems (and you can chose a sharable tiff compression method at
export time), all the above is probably a bit less of a necessity, and
I’m just making noise

A sharable tiff compression method is not trivial, it depends on the software and its capabilities used by the receiver.But this is a bit off-topic.

Markus M

Moritz

neteler · January 26, 2020, 10:50pm

Hi,

just for the record (by Markus M):

https://github.com/OSGeo/grass/pull/287

markusN