[GRASS-dev] [GRASS GIS] #3927: r.external vs r.in.gdal under Windows

#3927: r.external vs r.in.gdal under Windows
-----------------------------------+---------------------------------
Reporter: Nikos Alexandris | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone:
Component: Raster | Version: svn-releasebranch76
Keywords: r.external, r.in.gdal | CPU: Unspecified
Platform: MSWindows |
-----------------------------------+---------------------------------
This is a case of a (hard to trace) buggy behavior when linking an
external raster map of type Float32. Here an external report:
https://gitlab.com/natcapes/r.estimap.recreation/issues/36.

In short, a raster map of type Float32, linked to an EPSG:3035 Location,
via`r.external`, delivers different descriptive statistics under Windows
than what the identical workflow does under Linux (and what gdalinfo tells
about the data, both under Windows and Linux).

Also, the `--overwrite` flag given to either `r.external` or `r.in.gdal`
does not seem to work under Windows.

Perhaps all these are known issues?
Perhaps this case has to do with the presence of NULL cells too? As is the
case of the example map I use.

The example map `input_water_resources` is available at
https://gitlab.com/natcapes/r.estimap.recreation.data/blob/master/example_data_epsg_3035.tar.gz.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by Nikos Alexandris):

Maybe related somehow: https://trac.osgeo.org/grass/ticket/3857.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: major | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by hellik):

{{{
System Info
GRASS Version: 7.6.1
GRASS SVN revision: r74292
Build date: 2019-03-19
Build platform: x86_64-w64-mingw32
GDAL: 2.4.0
PROJ.4: 5.2.0
GEOS: 3.7.0
SQLite: 3.26.0
Python: 2.7.14
wxPython: 2.8.12.1
Platform: Windows-10-10.0.18362 (OSGeo4W)
}}}

{{{
D:\temp\yoga\example_data_epsg_3035\raster>gdalinfo
input_water_resources.tif -stats
Driver: GTiff/GeoTIFF
Files: input_water_resources.tif
Size is 2381, 2617
Coordinate System is:
PROJCS["ETRS89 / LAEA Europe",
     GEOGCS["ETRS89",
         DATUM["European_Terrestrial_Reference_System_1989",
             SPHEROID["GRS 1980",6378137,298.257222101,
                 AUTHORITY["EPSG","7019"]],
             TOWGS84[0,0,0,0,0,0,0],
             AUTHORITY["EPSG","6258"]],
         PRIMEM["Greenwich",0,
             AUTHORITY["EPSG","8901"]],
         UNIT["degree",0.0174532925199433,
             AUTHORITY["EPSG","9122"]],
         AUTHORITY["EPSG","4258"]],
     PROJECTION["Lambert_Azimuthal_Equal_Area"],
     PARAMETER["latitude_of_center",52],
     PARAMETER["longitude_of_center",10],
     PARAMETER["false_easting",4321000],
     PARAMETER["false_northing",3210000],
     UNIT["metre",1,
         AUTHORITY["EPSG","9001"]],
     AUTHORITY["EPSG","3035"]]
Origin = (4735600.000000000000000,2879700.000000000000000)
Pixel Size = (50.000000000000000,-50.000000000000000)
Metadata:
   AREA_OR_POINT=Area
   TIFFTAG_SOFTWARE=GRASS GIS 7.9.dev with GDAL 2.3.1
Image Structure Metadata:
   INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 4735600.000, 2879700.000) ( 15d39'28.78"E, 48d53'13.80"N)
Lower Left ( 4735600.000, 2748850.000) ( 15d31'39.85"E, 47d42'49.53"N)
Upper Right ( 4854650.000, 2879700.000) ( 17d16'27.34"E, 48d47'35.80"N)
Lower Right ( 4854650.000, 2748850.000) ( 17d 6'26.37"E, 47d37'20.45"N)
Center ( 4795125.000, 2814275.000) ( 16d23'29.04"E, 48d15'25.69"N)
Band 1 Block=2381x1 Type=Float32, ColorInterp=Gray
   Description = input_water_resources
   Minimum=0.000, Maximum=1.000, Mean=0.037, StdDev=0.103
   NoData Value=nan
   Metadata:
     COLOR_TABLE_RULES_COUNT=1
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.000000e+00 255 255 255 0 0 255
     STATISTICS_MAXIMUM=1.0000001192093
     STATISTICS_MEAN=0.037087096658156
     STATISTICS_MINIMUM=0
     STATISTICS_STDDEV=0.10325697665504
     STATISTICS_VALID_PERCENT=57.84
}}}

{{{
r.in.gdal
input=D:\temp\yoga\example_data_epsg_3035\raster\input_water_resources.tif
output=input_water_resources_ringdal
Importing raster map <input_water_resources_ringdal>...

g.region -p -a raster=input_water_resources_ringdal@data
align=input_water_resources_ringdal@data
projection: 99 (ETRS89 / LAEA Europe)
zone: 0
datum: etrs89
ellipsoid: grs80
north: 2879700
south: 2748850
west: 4735600
east: 4854650
nsres: 50
ewres: 50
rows: 2617
cols: 2381
cells: 6231077
}}}

{{{
r.univar -e map=input_water_resources_ringdal@data
total null and non-null cells: 6231077
total null cells: 2626801
Of the non-null cells:
----------------------
n: 3604276
minimum: 0
maximum: 1
range: 1
mean: 0.0370871
mean of absolute values: 0.0370871
standard deviation: 0.103257
variance: 0.010662
variation coefficient: 278.418 %
sum: 133672.132394677
1st quartile: 0
median (even number of cells): 0
3rd quartile: 0.000749337
90th percentile: 0.149497
}}}

{{{
r.external
input=D:\temp\yoga\example_data_epsg_3035\raster\input_water_resources.tif
output=input_water_resources_rexternal
Reading band 1 of 1...
Link to raster map <input_water_resources_rexternal> created.
}}}

{{{
r.univar -e map=input_water_resources_rexternal@data
total null and non-null cells: 6231077
total null cells: 5157779
Of the non-null cells:
----------------------
n: 1073298
minimum: 0.000101275
maximum: 1
range: 0.999899
mean: 0.124543
mean of absolute values: 0.124543
standard deviation: 0.157836
variance: 0.0249123
variation coefficient: 126.732 %
sum: 133672.132394677
1st quartile: 0.00224924
median (even number of cells): 0.0411837
3rd quartile: 0.234299
90th percentile: 0.329259
}}}

{{{
g.region -p -a raster=input_water_resources_rexternal@data
align=input_water_resources_rexternal@data
projection: 99 (ETRS89 / LAEA Europe)
zone: 0
datum: etrs89
ellipsoid: grs80
north: 2879700
south: 2748850
west: 4735600
east: 4854650
nsres: 50
ewres: 50
rows: 2617
cols: 2381
cells: 6231077
}}}

{{{
r.univar -e map=input_water_resources_rexternal@data
total null and non-null cells: 6231077
total null cells: 5157779
Of the non-null cells:
----------------------
n: 1073298
minimum: 0.000101275
maximum: 1
range: 0.999899
mean: 0.124543
mean of absolute values: 0.124543
standard deviation: 0.157836
variance: 0.0249123
variation coefficient: 126.732 %
sum: 133672.132394677
1st quartile: 0.00224924
median (even number of cells): 0.0411837
3rd quartile: 0.234299
90th percentile: 0.329259
}}}

>Perhaps all these are known issues? Perhaps this case has to do with the
presence of NULL >cells too? As is the case of the example map I use.

{{{
r.univar -e map=input_water_resources_ringdal@data
total null and non-null cells: 6231077
total null cells: 2626801
}}}

vs

{{{
r.univar -e map=input_water_resources_rexternal@data
total null and non-null cells: 6231077
total null cells: 5157779
}}}

null cells number seems to differ: r.in.gdal: 2626801 vs. r.external:
total null cells: 5157779

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------
Changes (by hellik):

* priority: major => critical

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by Nikos Alexandris):

Replying to [comment:2 hellik]:
>
> >Perhaps all these are known issues? Perhaps this case has to do with
the presence of NULL >cells too? As is the case of the example map I use.
>
>
> {{{
> r.univar -e map=input_water_resources_ringdal@data
> total null and non-null cells: 6231077
> total null cells: 2626801
> }}}
>
> vs
>
> {{{
> r.univar -e map=input_water_resources_rexternal@data
> total null and non-null cells: 6231077
> total null cells: 5157779
> }}}
>
> null cells number seems to differ: r.in.gdal: 2626801 vs. r.external:
total null cells: 5157779

This confirms my findings. I should have went for the `-e` flag too,
totally forgot it.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by Nikos Alexandris):

In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by hellik):

Replying to [comment:5 Nikos Alexandris]:
> In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425

why not fix it grass and let Qgis still use r.external?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by Nikos Alexandris):

Replying to [comment:6 hellik]:
> Replying to [comment:5 Nikos Alexandris]:
> > In-Directly related-affected: https://github.com/qgis/QGIS/pull/32425
>
> why not fix it grass and let Qgis still use r.external?

Sure, when the fix is there. Professional deadlines, however, cannot wait
for the real fix.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:7&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by Nikos Alexandris):

Replying to [comment:2 hellik]:
>
> {{{
> System Info
> GRASS Version: 7.6.1
> GRASS SVN revision: r74292
> Build date: 2019-03-19
> Build platform: x86_64-w64-mingw32
> GDAL: 2.4.0
> PROJ.4: 5.2.0
> GEOS: 3.7.0
> SQLite: 3.26.0
> Python: 2.7.14
> wxPython: 2.8.12.1
> Platform: Windows-10-10.0.18362 (OSGeo4W)
> }}}

Helmut, this is the result of a command, or did you handcraft-ed these
lines by copy-pasting?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:8&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by hellik):

Replying to [comment:8 Nikos Alexandris]:
> Replying to [comment:2 hellik]:
> >
> > {{{
> > System Info
> > GRASS Version: 7.6.1
> > GRASS SVN revision: r74292
> > Build date: 2019-03-19
> > Build platform: x86_64-w64-mingw32
> > GDAL: 2.4.0
> > PROJ.4: 5.2.0
> > GEOS: 3.7.0
> > SQLite: 3.26.0
> > Python: 2.7.14
> > wxPython: 2.8.12.1
> > Platform: Windows-10-10.0.18362 (OSGeo4W)
> > }}}
>
> Helmut, this is the result of a command, or did you handcraft-ed these
lines by copy-pasting?

Nothing hand-crafted, just go to the wxgui -> systeminfo

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:9&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by hellik):

Replying to [comment:9 hellik]:
> Replying to [comment:8 Nikos Alexandris]:
> > Replying to [comment:2 hellik]:
> > >
> > > {{{
> > > System Info
> > > GRASS Version: 7.6.1
> > > GRASS SVN revision: r74292
> > > Build date: 2019-03-19
> > > Build platform: x86_64-w64-mingw32
> > > GDAL: 2.4.0
> > > PROJ.4: 5.2.0
> > > GEOS: 3.7.0
> > > SQLite: 3.26.0
> > > Python: 2.7.14
> > > wxPython: 2.8.12.1
> > > Platform: Windows-10-10.0.18362 (OSGeo4W)
> > > }}}
> >
> > Helmut, this is the result of a command, or did you handcraft-ed these
lines by copy-pasting?
>
> Nothing hand-crafted, just go to the wxgui -> systeminfo

similar output by

{{{
g.version -r -e -g
version=7.8.1dev
date=2019
revision=4785a5b77
build_date=2019-10-25
build_platform=x86_64-w64-mingw32
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj=5.2.0
gdal=2.4.1
geos=3.8.0
sqlite=3.29.0
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:10&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone:
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by mmetz):

Apparently 0 (zero) and NULL (nodata) got confused in r.external on
Windows. If I set 0 (zero) to NULL (nodata), I get the exact same
incorrect results. No idea where and why this confusion in r.external on
Windows happens.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:11&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: critical | Milestone: 7.8.3
Component: Raster | Version: svn-releasebranch76
Resolution: | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------

Comment (by hellik):

fixed by

{{{

   Branch: refs/heads/master
   Home: https://github.com/OSGeo/grass
   Commit: 87c52954e206ab06d2aaea52b0398d0dd4fea783
https://github.com/OSGeo/grass/commit/87c52954e206ab06d2aaea52b0398d0dd4fea783
   Author: Markus Metz <33666869+metzm at users.noreply.github.com>
   Date: 2020-02-09 (Sun, 09 Feb 2020)

   Changed paths:
     M lib/raster/gdal.c

   Log Message:
   -----------
   r.external: read nan as nan (#338)

Fixes #329. atof() on windows can not read "nan" and returns 0 (zero)
instead. Using Rast_set_d_null_value() in this case to set nan directly.
}}}

{{{
   Branch: refs/heads/releasebranch_7_8
   Home: https://github.com/OSGeo/grass
   Commit: 607d3e126600c94ea31cbc8ff13974ad3e45a5b7
https://github.com/OSGeo/grass/commit/607d3e126600c94ea31cbc8ff13974ad3e45a5b7
   Author: Markus Metz <33666869+metzm at users.noreply.github.com>
   Date: 2020-02-09 (Sun, 09 Feb 2020)

   Changed paths:
     M lib/raster/gdal.c

   Log Message:
   -----------
   r.external: read nan as nan (#338)

Fixes #329. atof() on windows can not read "nan" and returns 0 (zero)
instead. Using Rast_set_d_null_value() in this case to set nan directly.
}}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:13&gt;
GRASS GIS <https://grass.osgeo.org>

#3927: r.external vs r.in.gdal under Windows
-------------------------------+-----------------------------------
  Reporter: Nikos Alexandris | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: critical | Milestone: 7.8.3
Component: Raster | Version: svn-releasebranch76
Resolution: fixed | Keywords: r.external, r.in.gdal
       CPU: Unspecified | Platform: MSWindows
-------------------------------+-----------------------------------
Changes (by hellik):

* status: new => closed
* resolution: => fixed

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3927#comment:14&gt;
GRASS GIS <https://grass.osgeo.org>