[Geoserver-devel] Missing netCDF and GRIB attributes on variables

Hello,

GeoServer 2.9.1 and most other recent versions seem to lose some netCDF (netcdf-extension) and GRIB (grib-extension) metadata on variables when the data is retrieved as netCDF (netcdf-out-extension). Note that I am using netCDF terminology throughout this email.

For example, in the gs-netcdf-out module (master/today) WCSNetCDFTest::testRapNativeGribRotatedPole() reads in the rap-native.grib2 file and writes it out as netCDF. I’ve attached the CDL of the input and output for each, and the most important information that is missing from the Temperature_surface variable in the output is:

  • The “coordinates” attribute
  • Grib_* attributes (useful to some consumers)
  • The “long_name” attribute
  • And it appears all attributes on variables other than “units”, “_FillValue”, and a few others

Global attributes (such as “Originating_or_generating_Center”) are also missing and important, but these can be configured for inclusion through the GeoServer web UI.

I am familiar with the netCDF Java API and happy to dig in to the code and submit a patch, if appropriate, but I am a new developer to GeoServer and it is unclear to me how these attributes on the variable might best be passed along to the NetCDFOutputManager. Setting GridCoverage2D “properties” might be a candidate but I want to check with those more familiar with the codebase to be sure.

Do you have any suggestions or cautions for adding this capability to GeoTools/GeoServer?

Thanks,
Aaron

(attachments)

rap-native-input.cdl (1.99 KB)
rap-native-output.cdl (1.43 KB)

Aaron,

I can confirm your findings. Note that the absence of a _FillValue implies that, as far as I know, the default for that data type is used. I have had user feedback that an explicit _FillValue is preferred to make the file more self documenting and to reinforce its use a missing value (and I think some also like a matching missing_value attribute).

There are more WCS NetCDF output problems:

- The GeoServer web UI lacks a means to set variable attributes for NetCDF output; only global attributes can be set.

- The dimension Units setting appears to be ignored; Unit Symbol being used instead.

- Custom dimension units are encoded as "FIXME" when not set.

- There is no way to set other attributes on dimensions, despite some of these being mandatory for CF-1.6 compliance, such the the positive attribute for vertical dimensions. The lack of attribute support is a killer when using a vertical sigma coordinate because there is nowhere to record the formula_terms that define it.

The dimension problems become more evident when using ImageMosaic to add extra dimensions. There is no point setting a global Conventions until the above issues are addressed and CF-1.6 compliance is possible.

Contributions to improve the fidelity and configurability of NetCDF output would be most welcome.

Kind regards,
Ben.

On 10/12/16 11:30, Aaron Braeckel wrote:

Hello,

GeoServer 2.9.1 and most other recent versions seem to lose some netCDF
(netcdf-extension) and GRIB (grib-extension) metadata on variables when the
data is retrieved as netCDF (netcdf-out-extension). Note that I am using
netCDF terminology throughout this email.

For example, in the gs-netcdf-out module (master/today)
WCSNetCDFTest::testRapNativeGribRotatedPole() reads in the rap-native.grib2
file and writes it out as netCDF. I've attached the CDL of the input and
output for each, and the most important information that is missing from
the Temperature_surface variable in the output is:

   - The "coordinates" attribute
   - Grib_* attributes (useful to some consumers)
   - The "long_name" attribute
   - And it appears all attributes on variables other than "units",
   "_FillValue", and a few others

Global attributes (such as "Originating_or_generating_Center") are also
missing and important, but these can be configured for inclusion through
the GeoServer web UI.

I am familiar with the netCDF Java API and happy to dig in to the code and
submit a patch, if appropriate, but I am a new developer to GeoServer and
it is unclear to me how these attributes on the variable might best be
passed along to the NetCDFOutputManager. Setting GridCoverage2D
"properties" might be a candidate but I want to check with those more
familiar with the codebase to be sure.

Do you have any suggestions or cautions for adding this capability to
GeoTools/GeoServer?

Thanks,
Aaron

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Thanks, Ben. I’ll see where what can be done with these netCDF issues. Can you suggest the most appropriate place for variable attribute information to be passed from the file reading step to the NetCDFOutputWriter? One candidate appears to be the Grid2DCoverage properties but I would like to hear from someone with better knowledge of the codebase.

Thanks,
Aaron

···

On Fri, Dec 9, 2016 at 8:21 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Aaron,

I can confirm your findings. Note that the absence of a _FillValue implies that, as far as I know, the default for that data type is used. I have had user feedback that an explicit _FillValue is preferred to make the file more self documenting and to reinforce its use a missing value (and I think some also like a matching missing_value attribute).

There are more WCS NetCDF output problems:

  • The GeoServer web UI lacks a means to set variable attributes for NetCDF output; only global attributes can be set.

  • The dimension Units setting appears to be ignored; Unit Symbol being used instead.

  • Custom dimension units are encoded as “FIXME” when not set.

  • There is no way to set other attributes on dimensions, despite some of these being mandatory for CF-1.6 compliance, such the the positive attribute for vertical dimensions. The lack of attribute support is a killer when using a vertical sigma coordinate because there is nowhere to record the formula_terms that define it.

The dimension problems become more evident when using ImageMosaic to add extra dimensions. There is no point setting a global Conventions until the above issues are addressed and CF-1.6 compliance is possible.

Contributions to improve the fidelity and configurability of NetCDF output would be most welcome.

Kind regards,
Ben.

On 10/12/16 11:30, Aaron Braeckel wrote:

Hello,

GeoServer 2.9.1 and most other recent versions seem to lose some netCDF
(netcdf-extension) and GRIB (grib-extension) metadata on variables when the
data is retrieved as netCDF (netcdf-out-extension). Note that I am using
netCDF terminology throughout this email.

For example, in the gs-netcdf-out module (master/today)
WCSNetCDFTest::testRapNativeGribRotatedPole() reads in the rap-native.grib2
file and writes it out as netCDF. I’ve attached the CDL of the input and
output for each, and the most important information that is missing from
the Temperature_surface variable in the output is:

  • The “coordinates” attribute
  • Grib_* attributes (useful to some consumers)
  • The “long_name” attribute
  • And it appears all attributes on variables other than “units”,
    “_FillValue”, and a few others

Global attributes (such as “Originating_or_generating_Center”) are also
missing and important, but these can be configured for inclusion through
the GeoServer web UI.

I am familiar with the netCDF Java API and happy to dig in to the code and
submit a patch, if appropriate, but I am a new developer to GeoServer and
it is unclear to me how these attributes on the variable might best be
passed along to the NetCDFOutputManager. Setting GridCoverage2D
“properties” might be a candidate but I want to check with those more
familiar with the codebase to be sure.

Do you have any suggestions or cautions for adding this capability to
GeoTools/GeoServer?

Thanks,
Aaron


Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi


Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Daniele and Simone,

because Atlassian Jira issue creation notifications are still down, here is Aaron's Jira issue:
https://osgeo-org.atlassian.net/browse/GEOS-7901

GeoTools pull request (still requires discussion):
https://github.com/geotools/geotools/pull/1417

Kind regards,
BEn.

On 13/12/16 04:49, Aaron Braeckel wrote:

Thanks, Ben. I'll see where what can be done with these netCDF issues.
Can you suggest the most appropriate place for variable attribute
information to be passed from the file reading step to the
NetCDFOutputWriter? One candidate appears to be the Grid2DCoverage
properties but I would like to hear from someone with better knowledge of
the codebase.

Thanks,
Aaron

On Fri, Dec 9, 2016 at 8:21 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Aaron,

I can confirm your findings. Note that the absence of a _FillValue implies
that, as far as I know, the default for that data type is used. I have had
user feedback that an explicit _FillValue is preferred to make the file
more self documenting and to reinforce its use a missing value (and I think
some also like a matching missing_value attribute).

There are more WCS NetCDF output problems:

- The GeoServer web UI lacks a means to set variable attributes for NetCDF
output; only global attributes can be set.

- The dimension Units setting appears to be ignored; Unit Symbol being
used instead.

- Custom dimension units are encoded as "FIXME" when not set.

- There is no way to set other attributes on dimensions, despite some of
these being mandatory for CF-1.6 compliance, such the the positive
attribute for vertical dimensions. The lack of attribute support is a
killer when using a vertical sigma coordinate because there is nowhere to
record the formula_terms that define it.

The dimension problems become more evident when using ImageMosaic to add
extra dimensions. There is no point setting a global Conventions until the
above issues are addressed and CF-1.6 compliance is possible.

Contributions to improve the fidelity and configurability of NetCDF output
would be most welcome.

Kind regards,
Ben.

On 10/12/16 11:30, Aaron Braeckel wrote:

Hello,

GeoServer 2.9.1 and most other recent versions seem to lose some netCDF
(netcdf-extension) and GRIB (grib-extension) metadata on variables when
the
data is retrieved as netCDF (netcdf-out-extension). Note that I am using
netCDF terminology throughout this email.

For example, in the gs-netcdf-out module (master/today)
WCSNetCDFTest::testRapNativeGribRotatedPole() reads in the
rap-native.grib2
file and writes it out as netCDF. I've attached the CDL of the input and
output for each, and the most important information that is missing from
the Temperature_surface variable in the output is:

   - The "coordinates" attribute
   - Grib_* attributes (useful to some consumers)
   - The "long_name" attribute
   - And it appears all attributes on variables other than "units",
   "_FillValue", and a few others

Global attributes (such as "Originating_or_generating_Center") are also
missing and important, but these can be configured for inclusion through
the GeoServer web UI.

I am familiar with the netCDF Java API and happy to dig in to the code and
submit a patch, if appropriate, but I am a new developer to GeoServer and
it is unclear to me how these attributes on the variable might best be
passed along to the NetCDFOutputManager. Setting GridCoverage2D
"properties" might be a candidate but I want to check with those more
familiar with the codebase to be sure.

Do you have any suggestions or cautions for adding this capability to
GeoTools/GeoServer?

Thanks,
Aaron

------------------------------------------------------------
------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand