#3622: r.out.gdal export in USGSDEM format takes very long time and finally
crashes
-------------------------------------------------+-------------------------
Reporter: dido | Owner: grass-dev@…
Type: defect | Status: new
Priority: critical | Milestone:
Component: Raster | Version: 7.4.0
Keywords: r.out.gdal export USGSDEM long time | CPU: x86-64
crashes |
Platform: MSWindows 7 |
-------------------------------------------------+-------------------------
I cannot attach any test raster because the one that manifests the problem
is very large. Here I apply raster metadata (I suppose any raster of this
size will do):
{{{
r.info map=BGMdem_8x8m_holes_filled_feathered@PERMANENT
+----------------------------------------------------------------------------+
| Map: BGMdem_8x8m_holes_filled_feat Date: Fri Aug 17 13:30:15 2018
|
| Mapset: PERMANENT Login of Creator: a
|
| Location: BG_DEM
|
| DataBase: E:\mapping\GRASS7DB
|
| Title: BGMdem_8x8m_holes_filled_feathered
|
| Timestamp: none
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 0
|
| Data Type: DCELL
|
| Rows: 52101
|
| Columns: 77901
|
| Total Cells: 4058720001
|
| Projection: UTM (zone 35)
|
| N: 4928368.63825695 S: 4511560.45143113 Res: 8.00000359
|
| E: 663749.15806402 W: 40538.62472039 Res: 8.00003252
|
| Range of data: min = -52.070711097653 max = 2925
|
|
|
| Data Description:
|
| generated by r.mapcalc
|
|
|
| Comments:
|
| if(isnull(defekti_rast_buf_recoded), BGMdem_8x8m_holes_filled,
|
| if(defekti_rast_buf_recoded < 100, BGMdem_8x8m_holes_filled * (100 -
|
| defekti_rast_buf_recoded) / 100 + ALOSdem_interp *
|
| defekti_rast_buf_recoded / 100, BGMdem_8x8m_holes_filled))
|
|
|
+----------------------------------------------------------------------------+
}}}
At crash time progress indicator has not advanced at all. Tool output
follows:
{{{
r.out.gdal -f --overwrite
input=BGMdem_8x8m_holes_filled_feathered@PERMANENT output=E:\tmp\BGM-dem-
f5p-csovp-wbf-cas-clgeo-s0_UTM_blended_20180816.dem format=USGSDEM
type=Int16
Driver <USGSDEM> does not support direct writing. Using MEM driver for
intermediate dataset.
WARNING: Precision loss: Raster map
<BGMdem_8x8m_holes_filled_feathered@PERMANENT> of type DCELL to be
exported as Int16. This can be avoided by using Float64.
WARNING: Forcing raster export
Checking GDAL data type and nodata value...
Using GDAL data type <Int16>
Input raster map contains cells with NULL-value (no-data). The value
-32768 will be used to represent no-data values in the input map. You can
specify a nodata value with the nodata option.
Exporting raster data to USGSDEM format...
}}}
From the message:
>Driver <USGSDEM> does not support direct writing. Using MEM driver for
intermediate >dataset.
Do you have enough memory to hold a 4 billion pixel raster in memory on
the computer?
See also https://www.gdal.org/frmt_mem.html
Not sure if either the MEM driver or the USGS DEM format can handle 4
billion pixels. Is there a reason that you can't use GeoTIFF ( up to 4GB
file Size)for the output file or GeoTIFF with the BIGTIFF option? ( up to
18,000 petabytes for a single file)
>WARNING: Precision loss: Raster map
<BGMdem_8x8m_holes_filled_feathered@PERMANENT> of >type DCELL to be
exported as Int16. This can be avoided by using Float64.
If you are going to truncate to integer, it might be better to use the
int() or round() functions in r.mapcalc prior to exporting.
#3622: r.out.gdal export in USGSDEM format takes very long time and finally
crashes
-------------------------+-------------------------------------------------
Reporter: dido | Owner: grass-dev@…
Type: defect | Status: new
Priority: critical | Milestone:
Component: Raster | Version: 7.4.0
Resolution: | Keywords: r.out.gdal export USGSDEM long time
| crashes
CPU: x86-64 | Platform: MSWindows 7
-------------------------+-------------------------------------------------
Comment (by dido):
GeoTIFF export runs fine, producing a 10G file. I chose to export to USGS
DEM because I expect a tenfold decrease in file size. At least that can be
observed if I do a DEM export of the same 10G GeoTIFF using e.g.
GlobalMapper.
I don't have an idea if the MEM driver will handle raster this big. My PC
has 16G of physical RAM and quite enough virtual memory.
Precision loss is fine for me as the input raster is also int16. I suppose
GRASS uses 64 bit floats internally (DCELL) and the truncation to int16
will be handled by the export driver. Actually loss of speed due to
truncation calculations is not an issue - GeoTIFF export runs for a quite
reasonable time.