[GRASS-user] skipping MEM driver with r.out.gdal for large files

GRASS GRASS user
,
1

Dear all,

Apologies, sent to the developer mailing list by mistake.

I am trying to save a rather large file from GRASS

Versions:
GRASS GIS 8.2.1
GDAL 3.6.3, released 2023/03/07
Linux: almalinux:9.1

The size of the file is:
rows=141120
cols=362880
cells=51209625600

This the command
r.out.gdal -c input=out output=output.tif format=COG type=Int16 createopt=COMPRESS=LZW,PREDICTOR=2,BIGTIFF=YES --overwrite --quiet

This the output
Warning 6: driver MEM does not support creation option COMPRESS
Warning 6: driver MEM does not support creation option PREDICTOR
Warning 6: driver MEM does not support creation option BIGTIFF
ERROR 2: /usr/gdal-3.6.2/frmts/mem/memdataset.cpp, 1289: cannot allocate 1x102419251200 bytes
ERROR: Unable to create dataset using memory raster driver

Is there any option not to use the MEM driver when writing to disk?
Thanks
Laura

2

Hi Laura,

On Thu, Jun 15, 2023 at 5:00 PM Laura Poggio <laura.poggio@gmail.com> wrote:

Dear all,
Apologies, sent to the developer mailing list by mistake.
I am trying to save a rather large file from GRASS

Versions:
GRASS GIS 8.2.1
GDAL 3.6.3, released 2023/03/07
Linux: almalinux:9.1

The size of the file is:
rows=141120
cols=362880
cells=51209625600

This the command
r.out.gdal -c input=out output=output.tif format=COG type=Int16 createopt=COMPRESS=LZW,PREDICTOR=2,BIGTIFF=YES --overwrite --quiet

This the output
Warning 6: driver MEM does not support creation option COMPRESS
Warning 6: driver MEM does not support creation option PREDICTOR
Warning 6: driver MEM does not support creation option BIGTIFF
ERROR 2: /usr/gdal-3.6.2/frmts/mem/memdataset.cpp, 1289: cannot allocate 1x102419251200 bytes
ERROR: Unable to create dataset using memory raster driver

Is there any option not to use the MEM driver when writing to disk?

I am not sure, in this code section it doesn't look like this:

https://github.com/OSGeo/grass/blob/51e962abbc1176211ad2c4cb7857a61a2b6d1c6f/raster/r.out.gdal/main.c#L377

Maybe someone else in the list has more insights?

Some options:
- try first the GTiff driver, then convert to COG with gdal_translate
- enlarge RAM using a swap file (no reboot needed but sudo access
needed) as disk space is relatively cheap and r.out.gdal will not
realize then that it gets swapped.

Best,
Markus

--
Markus Neteler, PhD
https://www.mundialis.de - free data with free software
https://grass.osgeo.org
https://courses.neteler.org/blog

3

Dear Markus,
Using GTiff (in grass version 8.3.0 and with gdal 3.7.1) solved the problem. I get a compressed file and the use of RAM is acceptable. I still get the same errors when using COG.
A quick test with a different file in grass 8.2.1 and gdal 3.6.4 (previous configuration) indicates that it would be the same there, but I could not test it with the original file because of some hardware issues.

Thanks a lot!

Laura

On Thu, 22 Jun 2023 at 09:16, Markus Neteler <neteler@osgeo.org> wrote:

Hi Laura,

On Thu, Jun 15, 2023 at 5:00 PM Laura Poggio <laura.poggio@gmail.com> wrote:

Dear all,
Apologies, sent to the developer mailing list by mistake.
I am trying to save a rather large file from GRASS

Versions:
GRASS GIS 8.2.1
GDAL 3.6.3, released 2023/03/07
Linux: almalinux:9.1

The size of the file is:
rows=141120
cols=362880
cells=51209625600

This the command
r.out.gdal -c input=out output=output.tif format=COG type=Int16 createopt=COMPRESS=LZW,PREDICTOR=2,BIGTIFF=YES --overwrite --quiet

This the output
Warning 6: driver MEM does not support creation option COMPRESS
Warning 6: driver MEM does not support creation option PREDICTOR
Warning 6: driver MEM does not support creation option BIGTIFF
ERROR 2: /usr/gdal-3.6.2/frmts/mem/memdataset.cpp, 1289: cannot allocate 1x102419251200 bytes
ERROR: Unable to create dataset using memory raster driver

Is there any option not to use the MEM driver when writing to disk?

I am not sure, in this code section it doesn’t look like this:

https://github.com/OSGeo/grass/blob/51e962abbc1176211ad2c4cb7857a61a2b6d1c6f/raster/r.out.gdal/main.c#L377

Maybe someone else in the list has more insights?

Some options:

  • try first the GTiff driver, then convert to COG with gdal_translate
  • enlarge RAM using a swap file (no reboot needed but sudo access
    needed) as disk space is relatively cheap and r.out.gdal will not
    realize then that it gets swapped.

Best,
Markus

–
Markus Neteler, PhD
https://www.mundialis.de - free data with free software
https://grass.osgeo.org
https://courses.neteler.org/blog