[Geoserver-devel] Handling null values in coverages

GeoServer GeoServer Developer
1

Hi,
I'm implementing the DescribeCoverage response for WCS 1.1
and I'm a little confused by no data handling.

First thing to notice is that WCS 1.1 only allows for
a list of single values, there is no allowance for a range
or doubles like the wcs 1.0 spec allowed for. This means
I cannot use the number range contained in CoverageDimension.

Looking at how CoverageDimension is built, it seems that
each CoverageDimension is in fact a rehashed GridSampleDimension.
Its null values are in fact grabbed for the one GridCategory
in the sample dimension whose name is "no data". So, what I
tried to do is to investigate and see how the range might have
been built in practice. Turns out that the only cases where
"no data" is used that I can find are in gtopo30 and arcgrid,
and are both single null values. GeoTiff seems to have no
support for null values (I guess this is just an implementation
limitation, right?). What about NetCDF and HDF, what should
I expect?

Yet GridCategory allows for a continuous range definition...
hum, where would that be used? These cases are not possible
to encode in WCS 1.1 as far as I can tell, so they would be
really troublesome...
Can anybody enlighten me?

Cheers
Andrea

2

Andrea Aime a écrit :

Its null values are in fact grabbed for the one GridCategory
in the sample dimension whose name is "no data".

This is not a reliable way to looks for the null value! However I realize that
we may need a convenience method making this job easier.

For a coverage of geophysics values (e.g. temperature, elevation, etc.), the
"null" values are anyone for which:

    Category.isQuantitative()

returns 'false'.

http://javadoc.geotools.fr/snapshot/org/geotools/coverage/Category.html#isQuantitative()

A geophysics coverage may have more than one null value.

For non-geophysics coverage (e.g. a coverage that contains only categories like
"forest", "river", "urban", etc.), all categories are non-quantitative and none
of them have special meaning. We could add some way to flag a particular
category as "nodata", but I would like to know why we would like to do so, in
order to see what could be the best approach. For example a possible approach
may be to check which pixel values in declared as transparent in the
IndexColorModel.

For "photographic" coverage, if we need a way to determine null values, then I
would like to know first what non-null values means in a photographic image...

Turns out that the only cases where
"no data" is used that I can find are in gtopo30 and arcgrid,
and are both single null values. GeoTiff seems to have no
support for null values (I guess this is just an implementation
limitation, right?). What about NetCDF and HDF, what should
I expect?

In NetCDF, the "fill values" are declared in the NetCDF metadata in some
specific place determined by the CF convention:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/ch02s05.html#missing-data

A NetCDF file can contains more than one "fill values". For those files, the
"Category.isQuantitative()" method call should be the most reliable way.

It should be the same for HDF data.

  Martin

3

Martin Desruisseaux ha scritto:

Andrea Aime a écrit :

Its null values are in fact grabbed for the one GridCategory
in the sample dimension whose name is "no data".

This is not a reliable way to looks for the null value!

That's the way the current WCS 1.0 implementation works, I'm just trying
to port it over to WCS 1.1.

However I realize that
we may need a convenience method making this job easier.

For a coverage of geophysics values (e.g. temperature, elevation, etc.), the
"null" values are anyone for which:

    Category.isQuantitative()

returns 'false'.

http://javadoc.geotools.fr/snapshot/org/geotools/coverage/Category.html#isQuantitative()

A geophysics coverage may have more than one null value.

Is each of these categories made up of a single value? As I said, with WCS 1.1 I'm apparently unable to model a continuous range of null values (like the closed interval [0.53, 10.5], just to make an example).

For non-geophysics coverage (e.g. a coverage that contains only categories like
"forest", "river", "urban", etc.), all categories are non-quantitative and none
of them have special meaning. We could add some way to flag a particular
category as "nodata", but I would like to know why we would like to do so, in
order to see what could be the best approach. For example a possible approach
may be to check which pixel values in declared as transparent in the
IndexColorModel.

hmm... WCS is about pure data, not about representation. Can we keep rendering issues away from the meaning of what is no-data and what is not? For example, rendering wise you might want to provide a solid color to the boave "forest", "river", "urban".

For "photographic" coverage, if we need a way to determine null values, then I
would like to know first what non-null values means in a photographic image...

For a photograpic coverage null values may represent the places where no image is available... yet I'm not sure one would have to report this one
as an explicit null value. Maybe so.

Turns out that the only cases where
"no data" is used that I can find are in gtopo30 and arcgrid,
and are both single null values. GeoTiff seems to have no
support for null values (I guess this is just an implementation
limitation, right?). What about NetCDF and HDF, what should
I expect?

In NetCDF, the "fill values" are declared in the NetCDF metadata in some
specific place determined by the CF convention:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.0/ch02s05.html#missing-data

A NetCDF file can contains more than one "fill values". For those files, the
"Category.isQuantitative()" method call should be the most reliable way.

It should be the same for HDF data.

I see, thank you :slight_smile:
Cheers
Andrea

4

Andrea Aime a écrit :

Is each of these categories made up of a single value? As I said, with
WCS 1.1 I'm apparently unable to model a continuous range of null values
(like the closed interval [0.53, 10.5], just to make an example).

In practice, yes a non-quantitative category is usually made of a single value.
The API allows a range of value however, but we could handle that in the same
way as a sequence of non-quantitative categories. Example:

  Category("Clouds", [0..3])

is that same than

  Category("Clouds", 0);
  Category("Clouds", 1);
  Category("Clouds", 2);
  Category("Clouds", 3);

since there is nothing preventing many categories to share the same name (only
the values must be differents).

  Martin

5

Martin Desruisseaux ha scritto:

Andrea Aime a écrit :

Is each of these categories made up of a single value? As I said, with
WCS 1.1 I'm apparently unable to model a continuous range of null values
(like the closed interval [0.53, 10.5], just to make an example).

In practice, yes a non-quantitative category is usually made of a single value.
The API allows a range of value however, but we could handle that in the same
way as a sequence of non-quantitative categories. Example:

  Category("Clouds", [0..3])

is that same than

  Category("Clouds", 0);
  Category("Clouds", 1);
  Category("Clouds", 2);
  Category("Clouds", 3);

since there is nothing preventing many categories to share the same name (only
the values must be differents).

So when the category is non quantitative the range min and max as always
integer numbers, and I can just generate the list of null values by
doing a for loop on them?

Cheers
Andrea

6

Andrea Aime a écrit :

So when the category is non quantitative the range min and max as always
integer numbers, and I can just generate the list of null values by
doing a for loop on them?

It is actually slightly simplier than that:

  Category.geophysics(false).getRange();

is always integer numbers (at least in practice - not sure about the theory, I
would have to refresh my mind), no matter if they are quantitative or not. So
you can do a loop on them.

For completness:

  Category.geophysics(true).getRange();

is the same range expressed in geophysics units (so usually floating point
numbers, but not necessarly).

  Martin