[GRASS-dev] Re: [gdal-dev] Setting NODATA to -nan

neteler · July 16, 2015, 1:04pm

Hi devs,

FWD of an issue with r.out.gdal in G7. Any ideas?

Markus

---------- Forwarded message ----------
From: Even Rouault <even.rouault@spatialys.com>
Date: Thu, Jul 16, 2015 at 2:07 AM
Subject: Fwd: Re: [gdal-dev] Setting NODATA to -nan
To: neteler@osgeo.org

Hi Markus,

Below you'll find the original report about what discussed on the way to the
party tonight, and then my analysis. The GDAL ticket with the dataset is
https://trac.osgeo.org/gdal/ticket/6036

I'm not sure there's really an issue in practice on the GDAL side. It is just
that dealing with nan is odd to anybody (and the way Windows and Linux
displays nan as a string is different).

Even

[gdal-dev] Setting NODATA to -nan
From: Homme Zwaagstra <hrz@geodata.soton.ac.uk>
To: gdal-dev <gdal-dev@lists.osgeo.org>
Date: Yesterday 11:21:54
Hello,

I've got the following raster exported from GRASS using `r.out.gdal`:

gdalinfo ppp-2015.tif
Driver: GTiff/GeoTIFF
Files: ppp-2015.tif
        ppp-2015.tif.aux.xml
Size is 432017, 216009
Coordinate System is:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.257223563,
             AUTHORITY["EPSG","7030"]],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0],
     UNIT["degree",0.0174532925199433],
     AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000833300541414,-0.000833298612558)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   COMPRESSION=DEFLATE
   INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)
Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
   NoData Value=nan
   Metadata:
     COLOR_TABLE_RULES_COUNT=5
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.681282e+03 255 255 0 0 255 0
     COLOR_TABLE_RULE_RGB_1=1.681282e+03 3.362563e+03 0 255 0 0 255 255
     COLOR_TABLE_RULE_RGB_2=3.362563e+03 5.043845e+03 0 255 255 0 0 255
     COLOR_TABLE_RULE_RGB_3=5.043845e+03 6.725127e+03 0 0 255 255 0 255
     COLOR_TABLE_RULE_RGB_4=6.725127e+03 8.406408e+03 255 0 255 255 0 0
     Generated_with=GRASS GIS 7.0.0

As you can see the NoData is set to `nan`. However, within the data it is
actually `-nan`:

gdallocationinfo ppp-2015.tif -wgs84 42.0776 30.9305
Report:
   Location: (266503P,70886L)
   Band 1:
     Value: -nan

I have tried to force the NoData to `-nan` as follows but with no joy:

gdal_translate -a_nodata '-nan' -of VRT ppp-2015.tif ppp-2015.vrt

gdalinfo ppp-2015.vrt
Driver: VRT/Virtual Raster
Files: ppp-2015.vrt
        /data/data/ppp_v2c_2015/ppp-2015.tif
Size is 432017, 216009
Coordinate System is:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.257223563,
             AUTHORITY["EPSG","7030"]],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0],
     UNIT["degree",0.0174532925199433],
     AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000833300541414,-0.000833298612558)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
   NoData Value=nan
   Metadata:
     COLOR_TABLE_RULES_COUNT=5
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.681282e+03 255 255 0 0 255 0
     COLOR_TABLE_RULE_RGB_1=1.681282e+03 3.362563e+03 0 255 0 0 255 255
     COLOR_TABLE_RULE_RGB_2=3.362563e+03 5.043845e+03 0 255 255 0 0 255
     COLOR_TABLE_RULE_RGB_3=5.043845e+03 6.725127e+03 0 0 255 255 0 255
     COLOR_TABLE_RULE_RGB_4=6.725127e+03 8.406408e+03 255 0 255 255 0 0
     Generated_with=GRASS GIS 7.0.0

Even editing the VRT to contain `<NoDataValue>-nan</NoDataValue>` yields a
positive nan value in gdalinfo.

This looks like it might be a GDAL bug to me? Is there a way I can specify
`-nan` as the NODATA value for this dataset without regenerating it
(it's quite
large)?

Many thanks,

Homme

---------- Forwarded Message ----------

Subject: Re: [gdal-dev] Setting NODATA to -nan
Date: Wednesday 15 July 2015, 20:02:11
From: Even Rouault <even.rouault@spatialys.com>
To: Homme Zwaagstra <hrz@geodata.soton.ac.uk>
CC: gdal-dev@lists.osgeo.org

Thanks Even,

I have submitted <https://trac.osgeo.org/gdal/ticket/6036> with an
attached TIFF
infested with `-nan` values.

While researching this I cam across this question
<http://stackoverflow.com/q/3772835> which might contain some information of
interest.

Hum looking more closely at representation of nan values in IEEE754, there is
at least 12 million different ways of representing nan for a 32bit floating
point value. And you can indeed have signed NaN... But I'm not sure it is
really useful to set "-nan" as the declared nodata value (it is stored as a
string in GeoTIFF) since even 2 NaN values that could be printed in text form
as "-nan" could have a different binary representation.

The parts of GDAL that test pixel values against nodata have normally a
special behaviour to test the pixel values against nodata, when the nodata
value is one of the NaN values, so I wouldn't expect this apparent
inconsistency to cause problems

For example when running gdalinfo -stats on your file, I get :

ERROR 1: negative-nan-example.tif, band 1: Failed to compute statistics, no
valid pixels found in sampling.

Which is expected since all pixels are set to nan (-nan yeah...), so
ComputeStatistics() actually managed to match the declared nan with the -nan
as pixel values. (which is quite ironical since a property of nan is that nan
!= nan, but here nan ~= -nan )

Did you run into particular problems beyond this apparent mismatch ?

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

-----------------------------------------
--
Spatialys - Geospatial professional services
http://www.spatialys.com

Yann4 · July 17, 2015, 11:02am

Hi,

it might be that some communication improvement from GRASS-GDAL could be done?

is there a clear NODATA/NAN definition understood within GDAL that we could use within r.out.gdal as a target NODATA value whenever anything than a number is used (i.e. NaN, nan, -nan, helloworld, mynameisbee, etc.)

typically MODIS users are used to -28768 and similar “standard” NODATA numbers, but that maybe another story.

0.01c

On Thu, Jul 16, 2015 at 6:34 PM Markus Neteler <neteler@osgeo.org> wrote:

Hi devs,

FWD of an issue with r.out.gdal in G7. Any ideas?

Markus

---------- Forwarded message ----------
From: Even Rouault <even.rouault@spatialys.com>
Date: Thu, Jul 16, 2015 at 2:07 AM
Subject: Fwd: Re: [gdal-dev] Setting NODATA to -nan
To: neteler@osgeo.org

Hi Markus,

Below you’ll find the original report about what discussed on the way to the
party tonight, and then my analysis. The GDAL ticket with the dataset is
https://trac.osgeo.org/gdal/ticket/6036

I’m not sure there’s really an issue in practice on the GDAL side. It is just
that dealing with nan is odd to anybody (and the way Windows and Linux
displays nan as a string is different).

Even

[gdal-dev] Setting NODATA to -nan
From: Homme Zwaagstra <hrz@geodata.soton.ac.uk>
To: gdal-dev <gdal-dev@lists.osgeo.org>
Date: Yesterday 11:21:54
Hello,

I’ve got the following raster exported from GRASS using r.out.gdal:

gdalinfo ppp-2015.tif
Driver: GTiff/GeoTIFF
Files: ppp-2015.tif
ppp-2015.tif.aux.xml
Size is 432017, 216009
Coordinate System is:
GEOGCS[“WGS 84”,
DATUM[“WGS_1984”,
SPHEROID[“WGS 84”,6378137,298.257223563,
AUTHORITY[“EPSG”,“7030”]],
AUTHORITY[“EPSG”,“6326”]],
PRIMEM[“Greenwich”,0],
UNIT[“degree”,0.0174532925199433],
AUTHORITY[“EPSG”,“4326”]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000833300541414,-0.000833298612558)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=DEFLATE
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0’ 0.00"W, 90d 0’ 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0’ 0.00"W, 90d 0’ 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0’ 0.00"E, 90d 0’ 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0’ 0.00"E, 90d 0’ 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0’ 0.01"E, 0d 0’ 0.01"N)
Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
NoData Value=nan
Metadata:
COLOR_TABLE_RULES_COUNT=5
COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.681282e+03 255 255 0 0 255 0
COLOR_TABLE_RULE_RGB_1=1.681282e+03 3.362563e+03 0 255 0 0 255 255
COLOR_TABLE_RULE_RGB_2=3.362563e+03 5.043845e+03 0 255 255 0 0 255
COLOR_TABLE_RULE_RGB_3=5.043845e+03 6.725127e+03 0 0 255 255 0 255
COLOR_TABLE_RULE_RGB_4=6.725127e+03 8.406408e+03 255 0 255 255 0 0
Generated_with=GRASS GIS 7.0.0

As you can see the NoData is set to nan. However, within the data it is
actually -nan:

gdallocationinfo ppp-2015.tif -wgs84 42.0776 30.9305
Report:
Location: (266503P,70886L)
Band 1:
Value: -nan

I have tried to force the NoData to -nan as follows but with no joy:

gdal_translate -a_nodata ‘-nan’ -of VRT ppp-2015.tif ppp-2015.vrt

gdalinfo ppp-2015.vrt
Driver: VRT/Virtual Raster
Files: ppp-2015.vrt
/data/data/ppp_v2c_2015/ppp-2015.tif
Size is 432017, 216009
Coordinate System is:
GEOGCS[“WGS 84”,
DATUM[“WGS_1984”,
SPHEROID[“WGS 84”,6378137,298.257223563,
AUTHORITY[“EPSG”,“7030”]],
AUTHORITY[“EPSG”,“6326”]],
PRIMEM[“Greenwich”,0],
UNIT[“degree”,0.0174532925199433],
AUTHORITY[“EPSG”,“4326”]]
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000833300541414,-0.000833298612558)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0’ 0.00"W, 90d 0’ 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0’ 0.00"W, 90d 0’ 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0’ 0.00"E, 90d 0’ 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0’ 0.00"E, 90d 0’ 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0’ 0.01"E, 0d 0’ 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
NoData Value=nan
Metadata:
COLOR_TABLE_RULES_COUNT=5
COLOR_TABLE_RULE_RGB_0=0.000000e+00 1.681282e+03 255 255 0 0 255 0
COLOR_TABLE_RULE_RGB_1=1.681282e+03 3.362563e+03 0 255 0 0 255 255
COLOR_TABLE_RULE_RGB_2=3.362563e+03 5.043845e+03 0 255 255 0 0 255
COLOR_TABLE_RULE_RGB_3=5.043845e+03 6.725127e+03 0 0 255 255 0 255
COLOR_TABLE_RULE_RGB_4=6.725127e+03 8.406408e+03 255 0 255 255 0 0
Generated_with=GRASS GIS 7.0.0

Even editing the VRT to contain <NoDataValue>-nan</NoDataValue> yields a
positive nan value in gdalinfo.

This looks like it might be a GDAL bug to me? Is there a way I can specify
-nan as the NODATA value for this dataset without regenerating it
(it’s quite
large)?

Many thanks,

Homme

---------- Forwarded Message ----------

Subject: Re: [gdal-dev] Setting NODATA to -nan
Date: Wednesday 15 July 2015, 20:02:11
From: Even Rouault <even.rouault@spatialys.com>
To: Homme Zwaagstra <hrz@geodata.soton.ac.uk>
CC: gdal-dev@lists.osgeo.org

Thanks Even,

I have submitted <https://trac.osgeo.org/gdal/ticket/6036> with an
attached TIFF
infested with -nan values.

While researching this I cam across this question
<http://stackoverflow.com/q/3772835> which might contain some information of
interest.

Hum looking more closely at representation of nan values in IEEE754, there is
at least 12 million different ways of representing nan for a 32bit floating
point value. And you can indeed have signed NaN… But I’m not sure it is
really useful to set “-nan” as the declared nodata value (it is stored as a
string in GeoTIFF) since even 2 NaN values that could be printed in text form
as “-nan” could have a different binary representation.

The parts of GDAL that test pixel values against nodata have normally a
special behaviour to test the pixel values against nodata, when the nodata
value is one of the NaN values, so I wouldn’t expect this apparent
inconsistency to cause problems

For example when running gdalinfo -stats on your file, I get :

ERROR 1: negative-nan-example.tif, band 1: Failed to compute statistics, no
valid pixels found in sampling.

Which is expected since all pixels are set to nan (-nan yeah…), so
ComputeStatistics() actually managed to match the declared nan with the -nan
as pixel values. (which is quite ironical since a property of nan is that nan
!= nan, but here nan ~= -nan )

Did you run into particular problems beyond this apparent mismatch ?

Even

–
Spatialys - Geospatial professional services
http://www.spatialys.com

gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

–
Spatialys - Geospatial professional services
http://www.spatialys.com

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Glynn_Clements1 · July 18, 2015, 10:48am

Yann Chemin wrote:

it might be that some communication improvement from GRASS-GDAL could be
done?

is there a clear NODATA/NAN definition understood within GDAL that we could
use within r.out.gdal as a target NODATA value whenever anything than a
number is used (i.e. NaN, nan, -nan, helloworld, mynameisbee, etc.)

typically MODIS users are used to -28768 and similar "standard" NODATA
numbers, but that maybe another story.

IMHO, anything using NaN as a no-data value should treat all NaNs
equally. The reason being that the only way to reliably propagate the
representation for a NaN is to memcpy() it. Assignment only preserves
NaN-ness, not necessarily the representation.

Actually, r.external is doing it wrong, as it expects any NaNs
supplied by GDAL to exactly match the representation generated by
Rast_set_d_null_value() (the all-ones bit pattern).

Fixed in r65602.

And if the no-data value isn't NaN, you have the issue of comparing
floating-point values for equality. Well, that will probably work so
long as the chosen no-data value is an integer.

--
Glynn Clements <glynn@gclements.plus.com>

martinl · July 18, 2015, 11:42am

Hi,

2015-07-18 12:48 GMT+02:00 Glynn Clements <glynn@gclements.plus.com>:

Fixed in r65602.

I took liberty to backport it to relbr70 in r65603. Martin

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa