[Geoserver-devel] Modifying Netcdf Output to include other variables

GeoServer GeoServer Developer
1

I have netcdf weather data that includes the following variables (I’ve left off the attributes and data):

byte wxVariable(t, z, y, x);

double t(t);

double z(z);

double y(y);

double z(z)

double start_time;

double stop_time;

double expiration_time;

double forecast_period(t);

double forecast_reference_time;

int grid_mapping;

The end user wants all of the ‘extra’ time variables included in the WCS netcdf output.

I can either alter Geoserver code to change the way the netcdf is written or I can modify the netcdf file outside of Geoserver before returning to the client.

First step is to determine if modifying Geoserver code to do this is possible within the current overall design.

I’ve been looking at the NetCDFOutputManager. Everything is based upon a GranuleStack associated with the wxVariable (assuming wxVariable is the CoverageId in a GetCoverage request). I don’t see a clean way to associate other variables within the netcdf data with the wxVariable.

Looking for options:

Is there a way to associate the other variables (start_time, stop_time, etc) with the wxVariable that simply overlooked?

Is there a clean way to have a GranuleStack include these extra variables? This idea seems incompatible with the current design, but figured I’d ask anyway.

Is there a clean way to include the original netcdf file path in the GranuleStack? If so, I could modify netcdf-out behavior to include ‘extra’ netcdf variables based upon configuration settings by reading the original file and extracting these ‘extra’ time variables and include them when writing the netcdf variable. I’m currently storing the netcdf file on disk. Not sure if this idea completely falls apart if data is stored differently.

Any suggestions is appreciated.

Thanks,

Jerry Wilwerding

2

Jerry,

I am looking at a similar problem. I would like a solution that also works with ImageMosaic to take a collection of scalar forecast_reference_time and output forecast_reference_time(t) as an extra variable alongside a data variable delivered by WCS.

This problem is also related to the need to gather GRIB/NetCDF variable attributes and add them to NetCDF output. See the analysis here:
https://github.com/geotools/geotools/pull/1417
https://osgeo-org.atlassian.net/browse/GEOS-7901

Again, the situation is more complicated when the output is created with ImageMosaic. I think I have found a spot in GranuleDescriptor where the required information can be passed with a small API change. I am investigating. My goal is to avoid duplication.

I suspect that we will need an API change, and thus a change proposal.

I see three things that we need:

(1) Collection of NetCDF/GRIB2 variable attributes from input and their application to WCS NetCDF output. What to do with ImageMosaic when attributes from different granules are mismatched is an open question.

(2) Identification of extra variables from input NetCDF/GRIB2 files for inclusion in WCS NetCDF output. This will include their attributes. We will probably have to discard attributes like "coordinates" as there is much renaming on output. This part will likely need a UI change.

(3) ImageMosaic of extra variables, for example forecast_reference_time mosaicked to forecast_reference_time(t). Whether this is generic or forecast_reference_time is handled as a special case is an open question.

Kind regards,
Ben.

On 02/03/17 01:02, Wilwerding, Jerry wrote:

I have netcdf weather data that includes the following variables (I've left off the attributes and data):

byte wxVariable(t, z, y, x);
double t(t);
double z(z);
double y(y);
double z(z)
double start_time;
double stop_time;
double expiration_time;
double forecast_period(t);
double forecast_reference_time;
int grid_mapping;

The end user wants all of the 'extra' time variables included in the WCS netcdf output.
I can either alter Geoserver code to change the way the netcdf is written or I can modify the netcdf file outside of Geoserver before returning to the client.
First step is to determine if modifying Geoserver code to do this is possible within the current overall design.

I've been looking at the NetCDFOutputManager. Everything is based upon a GranuleStack associated with the wxVariable (assuming wxVariable is the CoverageId in a GetCoverage request). I don't see a clean way to associate other variables within the netcdf data with the wxVariable.

Looking for options:
Is there a way to associate the other variables (start_time, stop_time, etc) with the wxVariable that simply overlooked?
Is there a clean way to have a GranuleStack include these extra variables? This idea seems incompatible with the current design, but figured I'd ask anyway.

Is there a clean way to include the original netcdf file path in the GranuleStack? If so, I could modify netcdf-out behavior to include 'extra' netcdf variables based upon configuration settings by reading the original file and extracting these 'extra' time variables and include them when writing the netcdf variable. I'm currently storing the netcdf file on disk. Not sure if this idea completely falls apart if data is stored differently.

Any suggestions is appreciated.

Thanks,

Jerry Wilwerding

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

3

And (2a), sometimes the extra variables might need to be full coverages themselves to ensure that output NetCDF files are fully self-describing. For example, if a coverage is dimensioned by a sigma vertical dimension, it might be necessary to include a pressure coverage to permit interpretation.

Kind regards,
Ben.

On 02/03/17 10:29, Ben Caradoc-Davies wrote:

Jerry,

I am looking at a similar problem. I would like a solution that also
works with ImageMosaic to take a collection of scalar
forecast_reference_time and output forecast_reference_time(t) as an
extra variable alongside a data variable delivered by WCS.

This problem is also related to the need to gather GRIB/NetCDF variable
attributes and add them to NetCDF output. See the analysis here:
https://github.com/geotools/geotools/pull/1417
https://osgeo-org.atlassian.net/browse/GEOS-7901

Again, the situation is more complicated when the output is created with
ImageMosaic. I think I have found a spot in GranuleDescriptor where the
required information can be passed with a small API change. I am
investigating. My goal is to avoid duplication.

I suspect that we will need an API change, and thus a change proposal.

I see three things that we need:

(1) Collection of NetCDF/GRIB2 variable attributes from input and their
application to WCS NetCDF output. What to do with ImageMosaic when
attributes from different granules are mismatched is an open question.

(2) Identification of extra variables from input NetCDF/GRIB2 files for
inclusion in WCS NetCDF output. This will include their attributes. We
will probably have to discard attributes like "coordinates" as there is
much renaming on output. This part will likely need a UI change.

(3) ImageMosaic of extra variables, for example forecast_reference_time
mosaicked to forecast_reference_time(t). Whether this is generic or
forecast_reference_time is handled as a special case is an open question.

Kind regards,
Ben.

On 02/03/17 01:02, Wilwerding, Jerry wrote:

I have netcdf weather data that includes the following variables (I've left off the attributes and data):

byte wxVariable(t, z, y, x);
double t(t);
double z(z);
double y(y);
double z(z)
double start_time;
double stop_time;
double expiration_time;
double forecast_period(t);
double forecast_reference_time;
int grid_mapping;

The end user wants all of the 'extra' time variables included in the WCS netcdf output.
I can either alter Geoserver code to change the way the netcdf is written or I can modify the netcdf file outside of Geoserver before returning to the client.
First step is to determine if modifying Geoserver code to do this is possible within the current overall design.

I've been looking at the NetCDFOutputManager. Everything is based upon a GranuleStack associated with the wxVariable (assuming wxVariable is the CoverageId in a GetCoverage request). I don't see a clean way to associate other variables within the netcdf data with the wxVariable.

Looking for options:
Is there a way to associate the other variables (start_time, stop_time, etc) with the wxVariable that simply overlooked?
Is there a clean way to have a GranuleStack include these extra variables? This idea seems incompatible with the current design, but figured I'd ask anyway.

Is there a clean way to include the original netcdf file path in the GranuleStack? If so, I could modify netcdf-out behavior to include 'extra' netcdf variables based upon configuration settings by reading the original file and extracting these 'extra' time variables and include them when writing the netcdf variable. I'm currently storing the netcdf file on disk. Not sure if this idea completely falls apart if data is stored differently.

Any suggestions is appreciated.

Thanks,

Jerry Wilwerding

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

4

On 02/03/17 01:02, Wilwerding, Jerry wrote:

Is there a clean way to include the original netcdf file path in the GranuleStack? If so, I could modify netcdf-out behavior to include 'extra' netcdf variables based upon configuration settings by reading the original file and extracting these 'extra' time variables and include them when writing the netcdf variable.

Jerry,

this is exactly what I plan to implement because it is the only solution that I have found that works with both single NetCDF/GRIB files and ImageMosaics of NetCDF/GRIB files. The GeoTools part of my implementation sets a SourceUrl property on each granule.

Another contributor found that properties set on coverages did not survive ImageMosiac because JAI trampled on them:
https://github.com/geotools/geotools/pull/1417

For my proposed changes, see:

GSIP 158 - NetCDF Output Support for Variable Attributes and Extra Variables
https://github.com/geoserver/geoserver/wiki/GSIP-158

The GeoTools part (SourceUrl) is done; no Jira or PR yet, until I am sure that it will work and I know whether I will need a SourceName property to support ImageMosaic coverage renaming:
https://github.com/bencaradocdavies/geotools/compare/master...coverage-source-url

The GSIP includes a link to the draft GeoServer UI and settings serialisation part. All up for discussion.

The UI is horrible but is really a placeholder; settings serialisation is the main goal. The NetCDFOutputManager implementation should be straightforward, except for expanding variables over dimensions which might be tricky.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

5

On 26/03/17 10:58, Ben Caradoc-Davies wrote:

On 02/03/17 01:02, Wilwerding, Jerry wrote:

Is there a clean way to include the original netcdf file path in the GranuleStack? If so, I could modify netcdf-out behavior to include 'extra' netcdf variables based upon configuration settings by reading the original file and extracting these 'extra' time variables and include them when writing the netcdf variable.

GSIP 158 - NetCDF Output Support for Variable Attributes and Extra Variables
https://github.com/geoserver/geoserver/wiki/GSIP-158

Jerry,

I have submitted pull requests for GSIP 158. This proposal includes support for "extra variables". The proposed implementation supports scalar source variables and vector output variables with values gathered from an ImageMosaic over a single non-spatial dimension.

GSIP 158 - NetCDF output support for variable attributes and extra variables
https://github.com/geoserver/geoserver/wiki/GSIP-158

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand