Hello everyone,
I not very expert in this field and searching on the web didn’t clarify my dubs.
I wanted to store a raster DEM and some vectors map in to an external DB, GRASS will use the DEM in order to calculate various raster map of solar radiation and need to store the again in the external DB. Than I’ll use r.mapcal to do some calculation of on the raster maps and i’ll use the vector map to extract this information. I don’t need to store the calculation I just need to perform the calculation and use the results immediately.
My question is which could be a feasible solution for this? how do I manage the fact that the input raster and vector map can have different projection? What about if I want to define a default projection of the DB and convert the maps when I load them in the DB?
The DB will probably be Postgis, do you have any other suggestion?
Thanks a lot
Lorenzo
Hi Lorenzo,
My suggestion is to not use PostGIS for big rasters, unless you have to, because you want to use the data in a specific application for example.
Loading raster to PG is a performace killer if you want to analyse the raster in GRASS (if that is possible at all)…
You can have the vector you want to calculate statistics on in PG. Load and reproject it to your DEM in GRASS, run your analysis and write the results (tables without geometry) back to PG and join them to your original vector map.
Do all spatial operations in GRASS (preferably raster based) meaning, convert vector to raster and use raster statistics (e.g. r.univar).
Cheers
Stefan
···
Hello everyone,
I not very expert in this field and searching on the web didn’t clarify my dubs.
I wanted to store a raster DEM and some vectors map in to an external DB, GRASS will use the DEM in order to calculate various raster map of solar radiation and need to store the again in the external DB. Than I’ll use r.mapcal to do some calculation of on the raster maps and i’ll use the vector map to extract this information. I don’t need to store the calculation I just need to perform the calculation and use the results immediately.
My question is which could be a feasible solution for this? how do I manage the fact that the input raster and vector map can have different projection? What about if I want to define a default projection of the DB and convert the maps when I load them in the DB?
The DB will probably be Postgis, do you have any other suggestion?
Thanks a lot
Lorenzo
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
...
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
# register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
# define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format="GTiff"
# perform calculations (here: extract pixels > 20 deg C)
# store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc "warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )"
# cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
# use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
The use of r.external makes a lot of sense when dealing with very large files. Does the use of “external” files work as expected in all of the new t.* modules?
Thanks,
Dylan
···
On Wed, Oct 7, 2015 at 12:09 PM, Markus Neteler <neteler@osgeo.org> wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
…
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format=“GTiff”
perform calculations (here: extract pixels > 20 deg C)
store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc “warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )”
cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Hi Dylan,
Yes, r.external and r.external.out works pretty well with TGIS. Here is a discussion in this regards I had mainly with Sören…
https://lists.osgeo.org/pipermail/grass-user/2013-December/069377.html
I have used r.external/r.external.out in TGIS for a while (with compressed GeoTiffs, btw you should not pre-define the predictor for LZW-compression as long as you are not sure that all raster maps are of the same type (CELL/FCELL/DCELL)), but went back to using internal GRASS format, because external links are a bit tricky to use on NFS and from different OS (or if data is moved):
https://trac.osgeo.org/grass/ticket/2660
Another minor (and often neglectable) drawback of using r.external is that file names and map names get out of sync if you rename maps. g.rename renames only the GRASS internal map name (modifies the link) the underlying data file is not touched and stays with the old name…
Cheers
Stefan
···
The use of r.external makes a lot of sense when dealing with very large files. Does the use of “external” files work as expected in all of the new t.* modules?
Thanks,
Dylan
On Wed, Oct 7, 2015 at 12:09 PM, Markus Neteler <neteler@osgeo.org> wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
…
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format=“GTiff”
perform calculations (here: extract pixels > 20 deg C)
store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc “warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )”
cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Hello,
Thanks for your suggestion. The point is that I need do develop a distributed infrastructure hence I need to store the raster in to a DB after the creation. Once I have created the raster immages I’ll need to do some simple multiplication and I thought to do them with r.mapcalc. Then I’ll use a vector map in order to extract the values of the raster.
So i think I’ll have to use inevitably a DBMS to store the raster images produced with grass. I’m I wright?
Tnx again regards
Lorenzo
···
2015-10-08 8:40 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:
Hi Dylan,
Yes, r.external and r.external.out works pretty well with TGIS. Here is a discussion in this regards I had mainly with Sören…
https://lists.osgeo.org/pipermail/grass-user/2013-December/069377.html
I have used r.external/r.external.out in TGIS for a while (with compressed GeoTiffs, btw you should not pre-define the predictor for LZW-compression as long as you are not sure that all raster maps are of the same type (CELL/FCELL/DCELL)), but went back to using internal GRASS format, because external links are a bit tricky to use on NFS and from different OS (or if data is moved):
https://trac.osgeo.org/grass/ticket/2660
Another minor (and often neglectable) drawback of using r.external is that file names and map names get out of sync if you rename maps. g.rename renames only the GRASS internal map name (modifies the link) the underlying data file is not touched and stays with the old name…
Cheers
Stefan
From: Dylan Beaudette [mailto:dylan.beaudette@gmail.com]
Sent: 8. oktober 2015 00:17
To: Markus Neteler <neteler@osgeo.org>
Cc: Blumentrath, Stefan <Stefan.Blumentrath@nina.no>; GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] External Database and projections
The use of r.external makes a lot of sense when dealing with very large files. Does the use of “external” files work as expected in all of the new t.* modules?
Thanks,
Dylan
On Wed, Oct 7, 2015 at 12:09 PM, Markus Neteler <neteler@osgeo.org> wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
…
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format=“GTiff”
perform calculations (here: extract pixels > 20 deg C)
store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc “warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )”
cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Hi Lorenzo,
What do you mean with a «distributed infrastructure”?
Do you want to make the rasters available via WWW or in your company/institution?
There are other options to share data than PostGIS… Within an organization your GRASS GIS database on NFS would be a much, much more efficient one.
For WWW-publishing WCS is an option…
Anyway, writing copies of your raster maps to DB (PostGIS) – if that is a fix requirement - is less of a problem (if they are not too heavy), than writing to DB, reading from DB for processing in GRASS and writing result back to DB…
If your aim is to add “zonal statistics” to a vector layer in PostGIS I would do the raster operation in GRASS and then join the result to PG…
Cheers
Stefan
···
Hello,
Thanks for your suggestion. The point is that I need do develop a distributed infrastructure hence I need to store the raster in to a DB after the creation. Once I have created the raster immages I’ll need to do some simple multiplication and I thought to do them with r.mapcalc. Then I’ll use a vector map in order to extract the values of the raster.
So i think I’ll have to use inevitably a DBMS to store the raster images produced with grass. I’m I wright?
Tnx again regards
Lorenzo
2015-10-08 8:40 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:
Hi Dylan,
Yes, r.external and r.external.out works pretty well with TGIS. Here is a discussion in this regards I had mainly with Sören…
https://lists.osgeo.org/pipermail/grass-user/2013-December/069377.html
I have used r.external/r.external.out in TGIS for a while (with compressed GeoTiffs, btw you should not pre-define the predictor for LZW-compression as long as you are not sure that all raster maps are of the same type (CELL/FCELL/DCELL)), but went back to using internal GRASS format, because external links are a bit tricky to use on NFS and from different OS (or if data is moved):
https://trac.osgeo.org/grass/ticket/2660
Another minor (and often neglectable) drawback of using r.external is that file names and map names get out of sync if you rename maps. g.rename renames only the GRASS internal map name (modifies the link) the underlying data file is not touched and stays with the old name…
Cheers
Stefan
From: Dylan Beaudette [mailto:dylan.beaudette@gmail.com]
Sent: 8. oktober 2015 00:17
To: Markus Neteler <neteler@osgeo.org>
Cc: Blumentrath, Stefan <Stefan.Blumentrath@nina.no>; GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] External Database and projections
The use of r.external makes a lot of sense when dealing with very large files. Does the use of “external” files work as expected in all of the new t.* modules?
Thanks,
Dylan
On Wed, Oct 7, 2015 at 12:09 PM, Markus Neteler <neteler@osgeo.org> wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
…
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format=“GTiff”
perform calculations (here: extract pixels > 20 deg C)
store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc “warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )”
cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
Hi Stefan,
With a distributed infrastructure I mean that the the computation and the data are deployed in a distributed platform. The data then will be available trough WWW but the aim of the infrastructure is not on the visualization of the maps but on providing the information in JSON or similar format regarding the information of the raster maps.
Cheers
Lorenzo
···
2015-10-14 16:09 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:
Hi Lorenzo,
What do you mean with a «distributed infrastructure”?
Do you want to make the rasters available via WWW or in your company/institution?
There are other options to share data than PostGIS… Within an organization your GRASS GIS database on NFS would be a much, much more efficient one.
For WWW-publishing WCS is an option…
Anyway, writing copies of your raster maps to DB (PostGIS) – if that is a fix requirement - is less of a problem (if they are not too heavy), than writing to DB, reading from DB for processing in GRASS and writing result back to DB…
If your aim is to add “zonal statistics” to a vector layer in PostGIS I would do the raster operation in GRASS and then join the result to PG…
Cheers
Stefan
From: Lorenzo Bottaccioli [mailto:lorenzo.bottaccioli@gmail.com]
Sent: 14. oktober 2015 15:44
To: Blumentrath, Stefan <Stefan.Blumentrath@nina.no>
Cc: Dylan Beaudette <dylan.beaudette@gmail.com>; Markus Neteler <neteler@osgeo.org>; GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] External Database and projections
Hello,
Thanks for your suggestion. The point is that I need do develop a distributed infrastructure hence I need to store the raster in to a DB after the creation. Once I have created the raster immages I’ll need to do some simple multiplication and I thought to do them with r.mapcalc. Then I’ll use a vector map in order to extract the values of the raster.
So i think I’ll have to use inevitably a DBMS to store the raster images produced with grass. I’m I wright?
Tnx again regards
Lorenzo
2015-10-08 8:40 GMT+02:00 Blumentrath, Stefan <Stefan.Blumentrath@nina.no>:
Hi Dylan,
Yes, r.external and r.external.out works pretty well with TGIS. Here is a discussion in this regards I had mainly with Sören…
https://lists.osgeo.org/pipermail/grass-user/2013-December/069377.html
I have used r.external/r.external.out in TGIS for a while (with compressed GeoTiffs, btw you should not pre-define the predictor for LZW-compression as long as you are not sure that all raster maps are of the same type (CELL/FCELL/DCELL)), but went back to using internal GRASS format, because external links are a bit tricky to use on NFS and from different OS (or if data is moved):
https://trac.osgeo.org/grass/ticket/2660
Another minor (and often neglectable) drawback of using r.external is that file names and map names get out of sync if you rename maps. g.rename renames only the GRASS internal map name (modifies the link) the underlying data file is not touched and stays with the old name…
Cheers
Stefan
From: Dylan Beaudette [mailto:dylan.beaudette@gmail.com]
Sent: 8. oktober 2015 00:17
To: Markus Neteler <neteler@osgeo.org>
Cc: Blumentrath, Stefan <Stefan.Blumentrath@nina.no>; GRASS user list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] External Database and projections
The use of r.external makes a lot of sense when dealing with very large files. Does the use of “external” files work as expected in all of the new t.* modules?
Thanks,
Dylan
On Wed, Oct 7, 2015 at 12:09 PM, Markus Neteler <neteler@osgeo.org> wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
…
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Here an example:
register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format=“GTiff”
perform calculations (here: extract pixels > 20 deg C)
store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc “warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )”
cease GDAL output connection and turn back to write standard GRASS
raster files:
r.external.out -r
use the result elsewhere
qgis $HOME/gisoutput/warm.tif
Best
Markus
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
On 07/10/15 21:09, Markus Neteler wrote:
On Wed, Oct 7, 2015 at 12:55 PM, Blumentrath, Stefan
<Stefan.Blumentrath@nina.no> wrote:
...
My suggestion is to not use PostGIS for big rasters, unless you have to,
because you want to use the data in a specific application for example.
Note r.external and r.external.out of GRASS GIS 7 for avoiding data duplication:
https://grass.osgeo.org/grass70/manuals/r.external.html
https://grass.osgeo.org/grass70/manuals/r.external.out.html
Note also that in PostGIS you can "store" raster data outside the database and only register a reference to the file and the necessary metadata in the database (cf the -R flag of raster2pgsql) [1].
That way you can access the same data from GRASS GIS and from PostGIS, avoiding imports and exports.
Moritz
[1] http://postgis.net/docs/manual-2.2/using_raster_dataman.html#RT_Raster_Loader