[GRASS-user] GRASS GIS virtual raster data sets

Dears,

I guess there is no GRASS GIS built-in implementation of GDAL's VRT
concept (several related ideas listed
https://trac.osgeo.org/grass/wiki/Grass8Planning).

Once past some compilation issues, I am about to try to build a VRT
based on native GRASS GIS raster maps (using GDAL with support for GRASS
GIS raster data), then link to this VRT via `r.external` from inside a
regular Mapset.

As a side-note, an image without pre-computed statistics stored in its
metadata, will require some time for `r.external` to work. The latter
likely forces this computation.

Perhaps it is faster to use `gdalinfo`'s '-stats' option to do this.

I wonder if there are smart ways to build, in GRASS GIS' data base,
a larger virtual raster map out of many smaller ones, without the need
to actually patch the latter ones in a new raster map file.

Nikos

On Wed, May 30, 2018 at 4:21 PM, Nikos Alexandris
<nik@nikosalexandris.net> wrote:

Dears,

I guess there is no GRASS GIS built-in implementation of GDAL's VRT
concept (several related ideas listed
https://trac.osgeo.org/grass/wiki/Grass8Planning).

I wish to see that as well (and discussed it a while ago with our dear
Markus M...

GRASS GIS would gain a lot having the possibility to virtually mosaik
raster tiles.

Once past some compilation issues, I am about to try to build a VRT
based on native GRASS GIS raster maps (using GDAL with support for GRASS
GIS raster data), then link to this VRT via `r.external` from inside a
regular Mapset.

Interesting, will that not cause loop dependencies?

I thought of true internal support for that to avoid loops.

As a side-note, an image without pre-computed statistics stored in its
metadata, will require some time for `r.external` to work. The latter
likely forces this computation.

Perhaps it is faster to use `gdalinfo`'s '-stats' option to do this.

I wonder if there are smart ways to build, in GRASS GIS' data base,
a larger virtual raster map out of many smaller ones, without the need
to actually patch the latter ones in a new raster map file.

I guess that some C code needs to be touched in lib/raster/ to allow
for a VRT-style raster concept.
Maybe we missed the current GSoC train for this idea :slight_smile:

markusN

On Thu, May 31, 2018 at 10:41 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 30, 2018 at 4:21 PM, Nikos Alexandris
<nik@nikosalexandris.net> wrote:

Dears,

I guess there is no GRASS GIS built-in implementation of GDAL’s VRT
concept (several related ideas listed
https://trac.osgeo.org/grass/wiki/Grass8Planning).

I wish to see that as well (and discussed it a while ago with our dear
Markus M…

I have an idea about how to implement native GRASS VRT, but no resources to do so…

GRASS GIS would gain a lot having the possibility to virtually mosaik
raster tiles.

Once past some compilation issues, I am about to try to build a VRT
based on native GRASS GIS raster maps (using GDAL with support for GRASS
GIS raster data), then link to this VRT via r.external from inside a
regular Mapset.

Interesting, will that not cause loop dependencies?

I thought of true internal support for that to avoid loops.

As a side-note, an image without pre-computed statistics stored in its
metadata, will require some time for r.external to work. The latter
likely forces this computation.

Perhaps it is faster to use gdalinfo’s ‘-stats’ option to do this.

It will not be faster because you would need gdalinfo -mm -stats in order to get the actual values, not an approximation. Sometimes an approximation of the stats based on a subsample is stored in the metadata of a raster.

I wonder if there are smart ways to build, in GRASS GIS’ data base,
a larger virtual raster map out of many smaller ones, without the need
to actually patch the latter ones in a new raster map file.

I guess that some C code needs to be touched in lib/raster/ to allow
for a VRT-style raster concept.

Yes.

Maybe we missed the current GSoC train for this idea :)>

The tricky part is to get a qualified student for this project.

Markus M

markusN


grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Hi,

Some experience with GDAL VRT and GRASS:

Be aware of: https://trac.osgeo.org/grass/ticket/2837

Also note also that esp. for integer rasters, compression of one big raster map can be much better than with lots of smaller tiles [1]. So I guess cases where patching by means of a virtual raster is most beneficial is in intermediate steps of an analysis chain…

Cheers

Stefan

1: https://www.mail-archive.com/grass-user@lists.osgeo.org/msg33125.html

···

On Thu, May 31, 2018 at 10:41 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 30, 2018 at 4:21 PM, Nikos Alexandris
<nik@nikosalexandris.net> wrote:

Dears,

I guess there is no GRASS GIS built-in implementation of GDAL’s VRT
concept (several related ideas listed
https://trac.osgeo.org/grass/wiki/Grass8Planning).

I wish to see that as well (and discussed it a while ago with our dear
Markus M…

I have an idea about how to implement native GRASS VRT, but no resources to do so…

GRASS GIS would gain a lot having the possibility to virtually mosaik
raster tiles.

Once past some compilation issues, I am about to try to build a VRT
based on native GRASS GIS raster maps (using GDAL with support for GRASS
GIS raster data), then link to this VRT via r.external from inside a
regular Mapset.

Interesting, will that not cause loop dependencies?

I thought of true internal support for that to avoid loops.

As a side-note, an image without pre-computed statistics stored in its
metadata, will require some time for r.external to work. The latter
likely forces this computation.

Perhaps it is faster to use gdalinfo’s ‘-stats’ option to do this.

It will not be faster because you would need gdalinfo -mm -stats in order to get the actual values, not an approximation. Sometimes an approximation of the stats based on a subsample is stored in the metadata of a raster.

I wonder if there are smart ways to build, in GRASS GIS’ data base,
a larger virtual raster map out of many smaller ones, without the need
to actually patch the latter ones in a new raster map file.

I guess that some C code needs to be touched in lib/raster/ to allow
for a VRT-style raster concept.

Yes.

Maybe we missed the current GSoC train for this idea :)>

The tricky part is to get a qualified student for this project.

Markus M

markusN


grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Essentially, I have "exactly", I think, the same workflow in mind. So, I am in a
path where you have been already. Thank you for sharing this Stefan.

- Computing LST images from Landsat 8.

- Instead of writing GRASS GIS native raster maps, I will switch to
  GeoTIFFs via r.external.out

- After having built time series for hundreds of WRS2 Paths and Rows
  combinations, I end up with some aggregated products.

- I tiled Europe in 10k^2 squares (in ETRS89) to have less tiles to deal
  with. Size of images per tile is acceptable (compressed via DEFLATE, ranging
  between less than 100 and up to 400MB).

- I "need" to avoid to patch these tiles in single European-wide raster
  maps. Thus the need, and the idea, for a "VRT".

So, I guess the idea won't work. I will do the attempt to "learn" this
by experience.

Thank you, Nikos

* Stefan Blumentrath <Stefan.Blumentrath@nina.no> [2018-06-01 07:09:08 +0000]:

Hi,

Some experience with GDAL VRT and GRASS:

Be aware of: https://trac.osgeo.org/grass/ticket/2837

Also note also that esp. for integer rasters, compression of one big raster map can be much better than with lots of smaller tiles [1]. So I guess cases where patching by means of a virtual raster is most beneficial is in intermediate steps of an analysis chain…

Cheers
Stefan

1: https://www.mail-archive.com/grass-user@lists.osgeo.org/msg33125.html

From: grass-user <grass-user-bounces@lists.osgeo.org> On Behalf Of Markus Metz
Sent: fredag 1. juni 2018 08:24
To: Markus Neteler <neteler@osgeo.org>
Cc: GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] GRASS GIS virtual raster data sets

On Thu, May 31, 2018 at 10:41 PM, Markus Neteler <neteler@osgeo.org<mailto:neteler@osgeo.org>> wrote:

On Wed, May 30, 2018 at 4:21 PM, Nikos Alexandris
<nik@nikosalexandris.net<mailto:nik@nikosalexandris.net>> wrote:
> Dears,
>
> I guess there is no GRASS GIS built-in implementation of GDAL's VRT
> concept (several related ideas listed
> https://trac.osgeo.org/grass/wiki/Grass8Planning).

I wish to see that as well (and discussed it a while ago with our dear
Markus M...

I have an idea about how to implement native GRASS VRT, but no resources to do so...

GRASS GIS would gain a lot having the possibility to virtually mosaik
raster tiles.

> Once past some compilation issues, I am about to try to build a VRT
> based on native GRASS GIS raster maps (using GDAL with support for GRASS
> GIS raster data), then link to this VRT via `r.external` from inside a
> regular Mapset.

Interesting, will that not cause loop dependencies?

I thought of true internal support for that to avoid loops.

> As a side-note, an image without pre-computed statistics stored in its
> metadata, will require some time for `r.external` to work. The latter
> likely forces this computation.
>
> Perhaps it is faster to use `gdalinfo`'s '-stats' option to do this.

It will not be faster because you would need gdalinfo -mm -stats in order to get the actual values, not an approximation. Sometimes an approximation of the stats based on a subsample is stored in the metadata of a raster.

>
> I wonder if there are smart ways to build, in GRASS GIS' data base,
> a larger virtual raster map out of many smaller ones, without the need
> to actually patch the latter ones in a new raster map file.

I guess that some C code needs to be touched in lib/raster/ to allow
for a VRT-style raster concept.

Yes.

Maybe we missed the current GSoC train for this idea :)>

The tricky part is to get a qualified student for this project.

Markus M

markusN
_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org<mailto:grass-user@lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/grass-user

_______________________________________________
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

--
Nikos Alexandris | Remote Sensing & Geomatics
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

* Markus Metz <markus.metz.giswork@gmail.com> [2018-06-01 08:24:20 +0200]:

On Thu, May 31, 2018 at 10:41 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 30, 2018 at 4:21 PM, Nikos Alexandris
<nik@nikosalexandris.net> wrote:
> Dears,
>
> I guess there is no GRASS GIS built-in implementation of GDAL's VRT
> concept (several related ideas listed
> https://trac.osgeo.org/grass/wiki/Grass8Planning).

I wish to see that as well (and discussed it a while ago with our dear
Markus M...

I have an idea about how to implement native GRASS VRT, but no resources to
do so...

That is great! If I may ask, is time the main requirement?

GRASS GIS would gain a lot having the possibility to virtually mosaik
raster tiles.

> Once past some compilation issues, I am about to try to build a VRT
> based on native GRASS GIS raster maps (using GDAL with support for GRASS
> GIS raster data), then link to this VRT via `r.external` from inside a
> regular Mapset.

Interesting, will that not cause loop dependencies?

I didn't think about it carefully then. Stefan's test shows that you are
right, I guess.

My naive thinking was: source images are, each, an inode. A VRT is a list of
filenames pointing to inodes. Link via r.external to the VRT which points
to the source inodes. Essentially, an indirect way to read GRASS GIS
raster maps, from inside a valid GRASS GIS session.

I guess I miss important parts of GRASS GIS' raster data model which
render the above schema to be of no use.

I thought of true internal support for that to avoid loops.

> As a side-note, an image without pre-computed statistics stored in its
> metadata, will require some time for `r.external` to work. The latter
> likely forces this computation.
>
> Perhaps it is faster to use `gdalinfo`'s '-stats' option to do this.

It will not be faster because you would need gdalinfo -mm -stats in order
to get the actual values, not an approximation. Sometimes an approximation
of the stats based on a subsample is stored in the metadata of a raster.

>
> I wonder if there are smart ways to build, in GRASS GIS' data base,
> a larger virtual raster map out of many smaller ones, without the need
> to actually patch the latter ones in a new raster map file.

I guess that some C code needs to be touched in lib/raster/ to allow
for a VRT-style raster concept.

Yes.

Maybe we missed the current GSoC train for this idea :)>

The tricky part is to get a qualified student for this project.

(I wish I could be in this position! :-/)

Nikos