[GRASS-dev] [GRASS GIS] #73: r.out.gdal tiff output does not work

#73: r.out.gdal tiff output does not work
------------------------------+---------------------------------------------
Reporter: helena | Owner: grass-dev@lists.osgeo.org
     Type: defect | Status: new
Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Keywords: r.out.gdal, tiff |
------------------------------+---------------------------------------------
r.out.gdal produces broken geo tiff files, see more details
in Dylan's notes and Eric's message, I am getting the same black area
with the nc_spm test data raster files. Only raster lakes that has two
colors
is readable.

http://casoilresource.lawr.ucdavis.edu/drupal/node/337

http://lists.osgeo.org/pipermail/grass-dev/2008-February/035987.html

Helena

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by hamish):

(I see the same with 6.2.1's old script version of r.out.gdal)

it works if the values are in the range of 0-255:
{{{
   #spearfish
   r.out.gdal in=fields out=fields.tif type=Byte
   qiv fields.tif # qiv is a quick image viewer
}}}

but not if all values are out of range:
{{{
   r.info -r elevation.dem
     min=1066
     max=1840
   r.info -t elevation.dem
     datatype=CELL
   r.out.gdal in=elevation.dem out=elv.tif type=Byte

   gdalinfo -stats elv.tif | grep STATISTIC
     STATISTICS_MINIMUM=255
     STATISTICS_MAXIMUM=255
     STATISTICS_MEAN=255
     STATISTICS_STDDEV=0

   r.in.gdal in=elv.tif out=elv2
   r.info -r elv2
     min=255
     max=255
}}}

we can try with half the raster in range:
{{{
   r.mapcalc "elev_0 = elevation.dem - 1066"
   r.info -r elev_0
     min=0
     max=774
   r.out.gdal in=elev_0 out=elv0.tif type=Byte
   gdalinfo -stats elv0.tif | grep STATISTIC
     STATISTICS_MINIMUM=0
     STATISTICS_MAXIMUM=254
     STATISTICS_MEAN=138.68943338231
     STATISTICS_STDDEV=56.526296226099
}}}

If I load elv0.tif into an image viewer it comes out as all black. But in
Gimp if you use the eyedropper you can see the index number changes, but
the palette is black for all 256 index entries.

gdalinfo confirms palette info was not written to file:

{{{
Band 1 Block=1899x4 Type=Byte, ColorInterp=Palette
   NoData Value=255
   Metadata:
     COLOR_TABLE_RULES_COUNT=0
   Color Table (RGB with 256 entries)
     0: 0,0,0,255
     1: 0,0,0,255
     2: 0,0,0,255
     3: 0,0,0,255
     4: 0,0,0,255
...
   255: 0,0,0,255
}}}

I do not know how to apply the Gimp's Topographic.gpl color palette to the
image's private "Colormap" index. It must be possible somehow, but I don't
know Gimp well enough..

If you want to store integer values >255 in the Tiff, you will have to use
one of Int16,UInt16,UInt32,Int32[,CInt16,CInt32?].

If you use type=Int32 etc you should preserve the CELL data for use in
other GIS software, even if the saved palette is bogus.

In the above elev_0 case you can see that the 0-774 raster exported as
Byte (8bit) only preserves elevation values in the range of 0-254.

Only Byte and UInt16 will attempt to write out a color palette to the
GeoTIFF AFAIK.

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

GRASS GIS wrote:

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by hamish):

(I see the same with 6.2.1's old script version of r.out.gdal)

it works if the values are in the range of 0-255:
{{{
   #spearfish
   r.out.gdal in=fields out=fields.tif type=Byte
   qiv fields.tif # qiv is a quick image viewer
}}}

but not if all values are out of range:
{{{
   r.info -r elevation.dem
     min=1066
     max=1840
   r.info -t elevation.dem
     datatype=CELL
   r.out.gdal in=elevation.dem out=elv.tif type=Byte

   gdalinfo -stats elv.tif | grep STATISTIC
     STATISTICS_MINIMUM=255
     STATISTICS_MAXIMUM=255
     STATISTICS_MEAN=255
     STATISTICS_STDDEV=0

   r.in.gdal in=elv.tif out=elv2
   r.info -r elv2
     min=255
     max=255
}}}

we can try with half the raster in range:
{{{
   r.mapcalc "elev_0 = elevation.dem - 1066"
   r.info -r elev_0
     min=0
     max=774
   r.out.gdal in=elev_0 out=elv0.tif type=Byte
   gdalinfo -stats elv0.tif | grep STATISTIC
     STATISTICS_MINIMUM=0
     STATISTICS_MAXIMUM=254
     STATISTICS_MEAN=138.68943338231
     STATISTICS_STDDEV=56.526296226099
}}}

If I load elv0.tif into an image viewer it comes out as all black. But in
Gimp if you use the eyedropper you can see the index number changes, but
the palette is black for all 256 index entries.

gdalinfo confirms palette info was not written to file:

I have noticed that r.out.gdal only generates the palette if
G_read_colors() returns 1, i.e. if the map has its own colour table.
If G_read_colors() falls back to a default rainbow colour table, you
don't get a palette.

I suggest:

--- raster/r.out.gdal/main.c (revision 30389)
+++ raster/r.out.gdal/main.c (working copy)
@@ -125,7 +125,7 @@
     GDALSetRasterColorInterpretation(hBand, GPI_RGB);
     CPLPopErrorHandler();

- if( G_read_colors(name, mapset, &sGrassColors ) == 1 )
+ if( G_read_colors(name, mapset, &sGrassColors ) >= 0 )
     {
   int maxcolor, i;
   CELL min, max;

G_read_colors() returns -1 on error (e.g. the map doesn't exist), 0 if
a default rainbow colour table was generated, and 1 if the map has its
own colour table.

AFAICT, the UInt16 case actually works, insofar as the debug output
with DEBUG=3 and the output from gdalinfo look just like they do for
an 8-bit TIFF which works fine in various viewers.

I wouldn't rule out the possibility that popular image viewers can't
handle 16-bit indexed-colour images.

TIFF's strong-point is that it supports so many different formats.
It's weak point is also that it supports so many different formats. I
doubt that there is a program in existence which can correctly handle
every possible valid TIFF file.

--
Glynn Clements <glynn@gclements.plus.com>

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by dylan):

The data range as reported in ArcMap 9.2 when opening a geotiff exported
from r.out.gdal is (1.17549e-038,3.40282e+038).

I have noticed that when setting the files symbology, when set to a fixed
number of classes (ranges) the correct data range is displayed, and the
image seems to work fine. This suggests that it may be an out of range or
unusual (in terms of ArcMap logic) no-data value.

[http://casoilresource.lawr.ucdavis.edu/drupal/node/337 I have updated my
notes here]

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by hamish):

Replying to [comment:2 dylan]:
> The data range as reported in ArcMap 9.2 when opening a geotiff exported
from
> r.out.gdal is (1.17549e-038,3.40282e+038).

If it were the NULL value, I'd expect to see extreme values only on one
side, not both. Endian problems is another suspect, but probably unlikely
if all on x86.

> I have noticed that when setting the files symbology, when set to a
fixed number
> of classes (ranges) the correct data range is displayed, and the image
seems to
> work fine. This suggests that it may be an out of range or unusual (in
terms of
> ArcMap logic) no-data value.

can you do like d.historgram or r.univar in Arc to see if the values are
streched over that range or are mostly correct but with weird outliers?

what if you try other output types besides Float32?

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by hamish):

(see also trac bug #67)

is there some other free software, either with or without GDAL that could
be used as a 3rd party test point?

Any info on what Arc treats as the no-data value? Does it respect what
GDAL puts in the metadata? Does Arc have the equivalent of r.null to clean
it up?

Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by helena):

Eric mentioned several viewers that don't like the tiff produced by
r.out.gdal and for me it won't work with Preview on Mac and gimp. You can
try for yourself:
here is an original tiff file that is readable by gimp and preview and
imports
to GRASS nc_spm_07 LOCATION using r.in.gdal (change tfwmf to tfw).
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/IMG3720079200P20040216.tfwmf
http://skagit.meas.ncsu.edu/~helena/grasswork/grassbookdat07/ncexternal/IMG3720079200P20040216.tif

After exporting with r.out.gdal the image is not readable. But you are
right that the numbers appear to be correct but there seem to be some out
of range values (nulls?). With some other rasters where I had nulls I
tried to define nulls as -9999 and export to AAGrid using r.out.gdal but
there was a problem too.
Here is one example of exported tiff
http://skagit.meas.ncsu.edu/~helena/grasswork/ortho_2001_t792_1m.tif
http://skagit.meas.ncsu.edu/~helena/grasswork/ortho_2001_t792_1m.tfw

there may be some option that exports the tiffs correctly but the only one
that was readable was this
http://skagit.meas.ncsu.edu/~helena/grasswork/lakes.tif
http://skagit.meas.ncsu.edu/~helena/grasswork/lakes.tfw

Thanks for looking into it

Helena

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Changes (by warmerdam):

* cc: warmerdam (added)

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
----------------------+-----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
----------------------+-----------------------------------------------------
Comment (by warmerdam):

Helena,

I'm coming into this rather late, and just skimming things. I looked at
your ortho_2001_t7921_1m.tif file (no smaller files demostrating the
problem!) and I see it is 32bit integer pixel values. I'm not surprised
lots of packages can't read this.

I looked at lakes.tif and it seems very similar to the ortho image in
configuration - also one 32bit integer sample. So I'm surprised that it
is readable and the ortho image is not in some (unnamed?) application.

PS. it would be convenient to post tiffinfo reports for some of the
problem files to learn more without having to download them. For instance
the ortho report is:

{{{
TIFF Directory at offset 0x23768d2 (37185746)
   Image Width: 3048 Image Length: 3048
   Bits/Sample: 32
   Sample Format: signed integer
   Compression Scheme: None
   Photometric Interpretation: min-is-black
   Samples/Pixel: 1
   Rows/Strip: 1
   Planar Configuration: single image plane
   Tag 33550: 1.000000,1.000000,0.000000
   Tag 33922:
0.000000,0.000000,0.000000,637033.000000,222504.000000,0.000000
   Tag 34735:
1,1,0,16,1024,0,1,1,1025,0,1,1,1026,34737,24,0,2048,0,1,4269,2049,34737,6,24,2054,0,1,9102,3072,0,1,32767,3074,0,1,32767,3075,0,1,8,3076,0,1,9001,3078,34736,1,2,3079,34736,1,3,3084,34736,1,1,3085,34736,1,0,3086,34736,1,4,3087,34736,1,5
   Tag 34736:
33.750000,-79.000000,36.166667,34.333333,609601.220000,0.000000
   Tag 34737: Lambert Conformal Conic|grs80|
   Tag 42112: <GDALMetadata>
   <Item name="COLOR_TABLE_RULES_COUNT" sample="0">1</Item>
   <Item name="COLOR_TABLE_RULE_RGB_0" sample="0">0.000000e+00 2.550000e+02
0 0 0 255 255 255</Item>
</GDALMetadata>
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>
GRASS Geographic Information System (GRASS GIS) - http://grass.osgeo.org/

#73: r.out.gdal tiff output does not work
-----------------------+----------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: default | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
-----------------------+----------------------------------------------------
Changes (by mlennert):

  * priority: major => critical

Comment:

We've been having similar problems here and I've tested a bit further with
current svn.

At http://geog-pc40.ulb.ac.be/grass/misc/r_out_gdal/ you can find relevant
files:

- test_loc.tgz: test location with one raster file
- test.tif: this same raster file exported with 'r.out.gdal type=UInt16'
- test_tiffinfo.txt: tiffinfo output on test.tif (note the errors at the
beginning)
- test _gdalinfo.txt gdalinfo output on test.tif, which shows the correct
color categories and then an all-black color table from 0 - 65535. Is it
normal to have both 'COLOR_TABLE_RULE_RGB_0' entries and a color table in
the tiff ?

The tif can be read and displayed by QGIS. SAGA gives a weird image (both
with band and pixel interleave): test_saga.png. gvSIG crashes when trying
to load the file. These are not some general picture viewer, but
cartography/GIS programs, so I would hope that we should be able to solve
this for reasons of interoperability.

Final test which seems to show that there is something wrong: r.in.gdal
in=test.tif out=test: import takes a very long time and creates a huge
colr table file:

% 0 65535
0:0
1:0
2:0
3:0
4:0
5:0
6:0
7:0
8:0
9:0
10:0
11:0
[...]
65535:0 65535:0

Whereas the original colr file only contains this:
% 334 460
334:0:191:191 359.2:0:255:0
359.2:0:255:0 384.4:255:255:0
384.4:255:255:0 409.6:255:127:0
409.6:255:127:0 434.8:191:127:63
434.8:191:127:63 460:20

The display of test also takes a long time, I suppose because of this long
colr table.

I'm moving this up to critical as I think this is a fairly basic feature
which should "just work"...

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

GRASS GIS wrote:

Final test which seems to show that there is something wrong: r.in.gdal
in=test.tif out=test: import takes a very long time and creates a huge
colr table file:

% 0 65535
0:0
1:0
2:0
3:0
4:0
5:0
6:0
7:0
8:0
9:0
10:0
11:0
[...]
65535:0 65535:0

Whereas the original colr file only contains this:
% 334 460
334:0:191:191 359.2:0:255:0
359.2:0:255:0 384.4:255:255:0
384.4:255:255:0 409.6:255:127:0
409.6:255:127:0 434.8:191:127:63
434.8:191:127:63 460:20

The display of test also takes a long time, I suppose because of this long
colr table.

r.out.gdal stores the colour table in a sequence of metadata items
named COLOR_TABLE_RULES_COUNT and COLOR_TABLE_RULE_RGB_<num>.

However, r.in.gdal ignores these settings. It just creates a colour
table which matches the palette of the imported file.

Although that's arguably the right thing to do]1], a 64K-entry table
is less than ideal. For a start, creating a colour table is O(n^2), as
each new entry is checked for overlap against all existing entries.

[1] If the data has been modified, the metadata items may have been
passed through untouched, so will no longer be correct.

One option is to provide an option to create the colour table from the
metadata items rather than from the palette. Another is to
"reverse-engineer" the colour table from the palette, replacing
sequences of palette entries with a single rule.

The latter is preferable, but it's non-trivial, particularly if you
consider that the original rules may have used non-integer endpoints.

--
Glynn Clements <glynn@gclements.plus.com>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Changes (by martinl):

  * platform: => Unspecified
  * component: default => Raster
  * cpu: => Unspecified

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by mmetz):

Replying to [comment:2 dylan]:
> The data range as reported in ArcMap 9.2 when opening a geotiff exported
from r.out.gdal is (1.17549e-038,3.40282e+038).

This perfectly right insofar as this is the potential data range for
float32. ArcMap does not check the actual data range when loading a new
layer the first time, even when real minimum and maximum values are given
in the metadata. The display of a layer gets more reasonable when choosing
e.g. histogram equalization as stretch because only now ArcMap is looking
at the real data range, gets real min and max values and adjusts the data
visualization accordingly.

ArcMap behaves like this too when opening ArcInfo ASCII grid and ESRI .hdr
labeled files with float32 data (same elevation raster exported into
different file formats, keeping data type float32). Maybe this is not a
GeoTIFF/GRASS/GDAL bug, but ArcMap is simply not very user-friendly with
data visualization?

Here, GRASS is much more user-friendly, because it always knows the data
range of a raster and, if no color table has been specified, automatically
generates a reasonable color table, usually in color and not grayscale as
ArcMap does by default...

The tiff file is handled by ArcMap all right, data and georeferencing
information are there, but data visualization needs to be modified by
hand.

>
> I have noticed that when setting the files symbology, when set to a
fixed number of classes (ranges) the correct data range is displayed, and
the image seems to work fine. This suggests that it may be an out of range
or unusual (in terms of ArcMap logic) no-data value.
>
> [http://casoilresource.lawr.ucdavis.edu/drupal/node/337 I have updated
my notes here]

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by mmetz):

Replying to [comment:1 hamish]:
> (I see the same with 6.2.1's old script version of r.out.gdal)
>
> it works if the values are in the range of 0-255:
>
> but not if all values are out of range:

It is equally right to say that the chosen datatype=Byte is not capable of
holding the full data range and a different datatype must be chosen.

> {{{
> r.info -r elevation.dem
> min=1066
> max=1840
> r.info -t elevation.dem
> datatype=CELL
> r.out.gdal in=elevation.dem out=elv.tif type=Byte
>
> r.in.gdal in=elv.tif out=elv2
> r.info -r elv2
> min=255
> max=255
> }}}
>
>
> we can try with half the raster in range:
> {{{
> r.mapcalc "elev_0 = elevation.dem - 1066"
> r.info -r elev_0
> min=0
> max=774
> r.out.gdal in=elev_0 out=elv0.tif type=Byte
No color table was specified with r.colors, so what data visualization
scheme is to be expected?
>
> If I load elv0.tif into an image viewer it comes out as all black. But
in Gimp if you use the eyedropper you can see the index number changes,
but the palette is black for all 256 index entries.

Gimp and many other image viewers only support 8 bit per channel,
cinepaint e.g. supports 32 bit. Most of them can not display tiff files
with anything else but Byte: only tiff images, not tiff datasets.
Elevation is a dataset, not an image. INT16 or UINT16 would be appropriate
to hold elevation data, in case of cm accuracy with meters as units,
FLOAT32 would be ok. Such tiff files hold a dataset. For elevation data,
GRASS offers several color tables which can be written to a tiff file as a
suggested data visualization scheme, if they have been set with r.colors.
Still, datatypes other than Byte can only be read with spatial data
viewers that could make use of this colour table, but not all of them are
actually able to do so. They could instead choose their own color schemes.
>
> gdalinfo confirms palette info was not written to file:
But the color palette was indeed written to the file.
>
> {{{
> Band 1 Block=1899x4 Type=Byte, ColorInterp=Palette
> NoData Value=255
> Metadata:
> COLOR_TABLE_RULES_COUNT=0
Below here. The instructions read to display each value with R:G:B = 0:0:0
= black. A data visualization scheme should have been provided with
r.colors to have control over the colortable written to the tiff file.
> Color Table (RGB with 256 entries)
> 0: 0,0,0,255
> 1: 0,0,0,255
> 2: 0,0,0,255
> 3: 0,0,0,255
> 4: 0,0,0,255
> ...
> 255: 0,0,0,255
> }}}
>

> If you want to store integer values >255 in the Tiff, you will have to
use one of Int16,UInt16,UInt32,Int32[,CInt16,CInt32?].
>
It would help a lot to have a list with these data types and their ranges
in the man page so that the appropriate data type for file export can be
chosen.

> If you use type=Int32 etc you should preserve the CELL data for use in
other GIS software, even if the saved palette is bogus.

Maybe the color table is not so important, different people have different
preferences e.g. for elevation data visualization.
If imagery is exported, e.g. after atmospheric correction or masking, the
datatype should probably be Byte as it was originally, e.g. to make
postprocessing in Gimp possible.
A tiff file that holds an image (e.g. GRASS display saved as tiff, a
scanned image, aerial photography, LANDSAT) does not have a color table,
usually. It does not need one, because the data in the file are already
R:G:B or grayscale (better: single band) values.

As long as both the data in the tiff file and its georeferencing are
identical to the original raster, the tiff file could be regarded as
perfectly ok. Suggested data visualization with an embedded color table is
nice to have, but coloring schemes are a matter of taste and not crucial
for further processing (e.g. terrain analysis with a DEM).

There is still the issue of nodata: a nodata value must fall into the
range of the chosen datatype, otherwise this information gets lost.
r.out.gdal does not check if a supplied nodata value is valid.

>
> Only Byte and UInt16 will attempt to write out a color palette to the
GeoTIFF AFAIK.
>
For UINT16, the color table in the tiff file will always have 65535
entries, gdal does that. This slows down display. Have an option in
r.out.gdal to choose whether a color table should be written to the tiff
file?

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by mmetz):

r.out.gdal GeoTIFF compatibility testing and new r.out.gdal test version

Changes in r.out.gdal test version
new flag for colortable export, default to no
nodata value adjustment to range of selected data type with warning
message, if user-specified nodata value is out of range
data range checking against the selected data type, abort with error if
range of selected data type exceeded
alignment of current region extends to raster resolution (original uses
current region extends and resolution), current region restored on exit
raster band statistics (min, max, mean, stddev) written to metadata if
supported, GDAL decides

I tested with QGIS-0.9.1 on Linux 64bit, QGIS -0.11.0-1 on XP, SAGA-2.0.3
on XP (doesn't work for me on Linux 64bit), gvSIG-1.1.2 (32bit binary) on
Linux 64bit, gvSIG-1.1.1 on XP, PCI Geomatica FreeView 9.1 on XP, ER
Viewer 7.2 (ER Mapper) on XP, and ArcMap 9.2 (ESRI) on XP.

Tests were performed with the elevation raster map layer (range within
Byte, FCELL map) in the North Carolina sample dataset, creating a MASK for
elevation values <= 70m, needed for testing of nodata handling with
nodata=0. The elevation raster was exported as Byte, UInt16, Int32, and
Float32, once with colortable (Byte and UInt16 only, colortable export for
other data types not supported by gdal), once without colortable, always
as GeoTIFF.

QGIS-0.9.1 on Linux 64bit

display: all ok

colortable: all ok

cell values: all ok

nodata: all ok

QGIS-0.11.1 on XP

display: all ok

colortable: all ok

cell values: all ok

nodata: all ok

gvSIG-1.1.2 (Linux 32bit binary) on Linux 64bit. Projection EPSG:32119

display: fail for UInt16 with colortable, all other ok

colortable: fail for UInt16, Byte ok

cell values: fail for Byte!!! (sometimes reports negative values), all
other ok. Interestingly the display seems to use correct values. Bug in
gvSIG-1.1.2

nodata: nodata not supported, actual cell value reported

gvSIG-1.1.1 on XP, Projection EPSG:32119

display: fail for UInt16 with colortable, all other ok

colortable: fail for Uint16, ok for Byte

cell values: fail for Byte!!! (sometimes reports negative values), all
other ok. Interestingly the display seems to use correct values. Bug in
gvSIG-1.1.1

nodata: nodata not supported, actual cell value reported

PCI Geomatica FreeView 9.1 on XP

display: all ok

colortable: all ok

cell values: all ok

nodata: nodata not supported, actual cell value reported

ER Viewer 7.2 (ER Mapper) on XP

display: fail for UInt16 with colortable, all other ok

colortable: fail for UInt16, ok for Byte

cell values: cell value query not supported

nodata: cell value query not supported

SAGA-2.0.3 on XP (I don't get it to run on Linux 64bit)

display: all ok

colortable: all ignored

cell values: all ok

nodata: all ok

ESRI ArcMap 9.2 on XP

display: all ok, but all without colortable and other than Byte need
manual adjustment which is painfully slow, even for these small raster
layers (Properties -> Symbology -> Stretch -> Minimum-Maximum)

colortable: all ok

cell values: all ok

nodata: all ok

Colortables are not allways properly rendered. GIMP and image viewers
which are not using GDAL render a colortable for Byte type properly. Data
types other than Byte can not be read by these 'standard' image editing
and viewing applications.

If nodata specification is not supported, applications return the actual
cell value, in this case 0, instead of "NoData", "NULL", "" or similar.
The files are still perfectly readable, only nodata assignment doesn't
happen.

I suggest adding a flag for colortable export with default to no and
nodata value checking as well as data range checking against the selected
data type.

Another issue I discovered is that r.out.gdal uses the current region
settings for export. This can cause implicit resampling to a different
resolution which should be avoided IMHO. Instead, r.out.gdal could use the
current region extends, but align the region settings used for export to
the raster to be exported. This way, a subregion can still be exported,
but the resolution of the GRASS raster is preserved. Also maybe check
whether the current region extends are within the raster extends?
Alignment of the current region to the raster to be exported by a new
r.out.gdal is not crucial, it can be done manually beforehand with
g.region align="myraster", but it would be very convenient if r.out.gdal
does that and restores the original region settings after successful
export.

The test version of gdal is available at
http://markus.metz.giswork.googlepages.com/r.out.gdal.conservative.tar.gz

See enclosed README for full description and justification of all changes.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

Great work!
Would you mind to also test back-import into GRASS?

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:14&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by mmetz):

Replying to [comment:14 neteler]:
> Great work!
Thanks!
> Would you mind to also test back-import into GRASS?
All GeoTIFFs previously exported with the test version of r.out.gdal were
back-imported to GRASS-6.4.svn (snapshot_2008_08_23)

GeoTIFF files:[[BR]]
elevation.Byte.clrtbl.tif[[BR]]
elevation.Byte.noclrtbl.tif[[BR]]
elevation.UInt16.clrtbl.tif[[BR]]
elevation.UInt16.noclrtbl.tif[[BR]]
elevation.Int32.noclrtbl.tif[[BR]]
elevation.Float32.noclrtbl.tif

display: all ok, very slow for the UInt16 back-import with
colortable[[BR]]
colortable: all ok[[BR]]
cell values: all ok[[BR]]
nodata: all ok

Back-importing elevation.UInt16.clrtbl.tif is pretty slow. Display takes
quite some time too. It is a colortable with 65535 entries, one for each
potential cell value, an exact copy of the GeoTIFF colortable. Removing
the colortable with r.colors -r, then copying the colortable of the back-
import of elevation.Byte.clrtbl.tif results in the same display, but now
fast on refresh, as expected.

Back-import of all other GeoTIFF files is fast, as is display in GRASS.

I would prefer to use r.out.gmt to get the GRASS color rules into an extra
file (GMT .cpt). This would also be a workaround for export file types
other than Byte and UInt16 where GDAL does not support writing out a
colortable. But then I don't know how widely GMT .cpt files are supported.

Markus

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/73#comment:15&gt;
GRASS GIS <http://grass.osgeo.org>

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

Markus (Metz),

what about integrating your fixes from
  http://markus.metz.giswork.googlepages.com/r.out.gdal.conservative.tar.gz
?

Markus

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/73#comment:16&gt;
GRASS GIS <http://grass.osgeo.org>

GRASS GIS wrote:

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
  Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by neteler):

Markus (Metz),

what about integrating your fixes from
  http://markus.metz.giswork.googlepages.com/r.out.gdal.conservative.tar.gz
?

Markus

Sure, no objections from my side, I'm using this version only. But
r.out.gdal is a very important module of GRASS and maybe some more
testing is required. Also, the new features may not find approval by
all, e.g. I've put in rather restrictive NULL cell handling: the user
has to specify a nodata value that falls within the range of the
selected datatype (Byte, UInt16, Int16, ...), a nodata value is no
longer chosen automatically as in the original version. If a nodata
value is not given, but NULL cells are present, r.out.gdal aborts with
an error message. My version also no longer uses the current region
resolution, instead the current region extends are aligned to the
resolution of the raster to be exported to avoid any implicit
resampling. And the colortable is only exported on request, and then
with a warning message.
If these changes are ok with you then integrate the changes, otherwise
maybe only some, but not all changes could be integrated. More
discussion needed on how to change/improve r.out.gdal?

Markus

On 24/10/08 14:10, Markus Metz wrote:

GRASS GIS wrote:

#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
  Reporter: helena | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new Priority: critical | Milestone: 6.4.0 Component: Raster | Version: svn-trunk Resolution: | Keywords: r.out.gdal, tiff Platform: Unspecified | Cpu: Unspecified --------------------------+-------------------------------------------------
Comment (by neteler):

Markus (Metz),

what about integrating your fixes from
  http://markus.metz.giswork.googlepages.com/r.out.gdal.conservative.tar.gz
?

Markus

Sure, no objections from my side, I'm using this version only. But
r.out.gdal is a very important module of GRASS and maybe some more
testing is required. Also, the new features may not find approval by
all, e.g. I've put in rather restrictive NULL cell handling: the user
has to specify a nodata value that falls within the range of the
selected datatype (Byte, UInt16, Int16, ...), a nodata value is no
longer chosen automatically as in the original version. If a nodata
value is not given, but NULL cells are present, r.out.gdal aborts with
an error message. My version also no longer uses the current region
resolution, instead the current region extends are aligned to the
resolution of the raster to be exported to avoid any implicit
resampling. And the colortable is only exported on request, and then
with a warning message.
If these changes are ok with you then integrate the changes, otherwise
maybe only some, but not all changes could be integrated. More
discussion needed on how to change/improve r.out.gdal?

+1 for all of these changes. Let's try with those and then discuss.

Moritz