Hei,
Recently I conducted some solar radiation analysis at national scale. In order to run this efficiently, I tiled the area into ca. 590 tiles of up to 5000x5000 pixels for the relevant (land) area. For reducing the amount of data I truncated the r.sun output to integer with r.mapcalc (map type is now CELL) and the size of the tiles is up to 25MB each.
Currently I am patching the tiles together by day (r.patch). However, this is
a) time consuming and
b) patching the sparse tiles into a coherent raster map roughly triples the amount of storage occupied (due to the relatively large number of NoData pixels in the patched raster map I guess). One patched map is 48 GB, while the sum of the tiles probably is ca. 15 GB. I will try to see which effect NULL-map compression in G7.2 has, but I doubt that it will bring back the storage efficiency of the tiles…
Therefore, I was wondering if it would be of interest and possible to implement a concept similar to GDAL VRT in GRASS (8?)?
A workaround would probably be to use r.external.out (write out tiles to e.g. GeoTIFF) in combination with gdal_buildvrt and r.external. But I would prefer to stick to native GRASS format (amongst others because then data management is more simple in a work flow which is parallelized by day within one single mapset)… I earlier tried to link a GDAL VRT from GRASS data (having the GDAL-GRASS-plugin installed) back to GRASS but encountered the issue that data was written to the mapset where the tiles reside (instead of the current mapset): https://trac.osgeo.org/grass/ticket/2837
No idea how the performance of a linked VRT consisting of hundreds of LZW-compressed GeoTiffs would be…
I would be happy about any hints, comments, or opinions…
Kind regards,
Stefan
Hi again,
It seems that in GRASS 7.3 (r68881) – which I used for patching - compression was not fully implemented.
I am recompressing with latest GRASS 7.2 and
export GRASS_INT_ZLIB=7
export GRASS_COMPRESSOR=ZLIB
export GRASS_COMPRESS_NULLS=1
now and the files are compressed to ca. 9 GB (ca. 78% compressed) and thus significantly smaller than the sum of the tiles…
Cheers
Stefan
···
From: grass-user [mailto:grass-user-bounces@lists.osgeo.org] On Behalf Of Blumentrath, Stefan
Sent: 23. oktober 2016 21:08
To: grass-user grass-user (grass-user@lists.osgeo.org) grass-user@lists.osgeo.org
Subject: [GRASS-user] Virtual mosaics in GRASS?
Hei,
Recently I conducted some solar radiation analysis at national scale. In order to run this efficiently, I tiled the area into ca. 590 tiles of up to 5000x5000 pixels for the relevant (land) area. For reducing the amount of data I truncated the r.sun output to integer with r.mapcalc (map type is now CELL) and the size of the tiles is up to 25MB each.
Currently I am patching the tiles together by day (r.patch). However, this is
a) time consuming and
b) patching the sparse tiles into a coherent raster map roughly triples the amount of storage occupied (due to the relatively large number of NoData pixels in the patched raster map I guess). One patched map is 48 GB, while the sum of the tiles probably is ca. 15 GB. I will try to see which effect NULL-map compression in G7.2 has, but I doubt that it will bring back the storage efficiency of the tiles…
Therefore, I was wondering if it would be of interest and possible to implement a concept similar to GDAL VRT in GRASS (8?)?
A workaround would probably be to use r.external.out (write out tiles to e.g. GeoTIFF) in combination with gdal_buildvrt and r.external. But I would prefer to stick to native GRASS format (amongst others because then data management is more simple in a work flow which is parallelized by day within one single mapset)… I earlier tried to link a GDAL VRT from GRASS data (having the GDAL-GRASS-plugin installed) back to GRASS but encountered the issue that data was written to the mapset where the tiles reside (instead of the current mapset): https://trac.osgeo.org/grass/ticket/2837
No idea how the performance of a linked VRT consisting of hundreds of LZW-compressed GeoTiffs would be…
I would be happy about any hints, comments, or opinions…
Kind regards,
Stefan