RE: [Geoserver-devel] Re: [Geoserver-users] netcdf - wcs resource

GeoServer GeoServer Developer
1

Hi Rob,

am currently trying to organise some effort to get a major
new geotools enhancement (Complex features) into the
Geoserver head, and the WCS support is an obvious target to
try to roll in. Would be grateful if you can identify any
issues, futher development work required to roll that in, and
also if any committed effort for this can be rolled into my
project plan as in-kind support. This sort of in-kind support
will help make the case for the bits of cash we are chasing
to support UI upgrades for the new stuff etc.

Usual story - really want this, but no effort to put in... :frowning:

Is there a UI config for the new WCS stuff? If not, what
needs to be done?

If you stick with a small subset (perhaps one) of netCDF conventions you
can imagine a netCDF collection 'scanner' that picks out e.g.
spatiotemporal metadata defined by those conventions across the fileset,
and presents additional configurable options in a GUI. We're working on
this in NERC DataGrid for CSML - rather than geoserver config -
generation.

In general - we have two requirements in the medium term:
1) auto-configure WCS for a catalogue of NetCDF files, to
support subsetting actions easily having found a netCDF file
in a catalogue

In most atmos/ocean use cases you don't want file-level discovery. You
want the aggregated (timeseries) dataset.

2) be able to subset (in a single WCS call) from collections
of netCDF files across both spatial and temporal dimensions.

Yep.

Also, there are many netCDF conventions - what is the
expected level of support and configurability?

The 'CF' conventions for atmos/ocean data are the canonical standard:
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html
though (as far as I'm aware) not implemented in full *anywhere*.
However, an incremental approach can be taken - e.g. starting with
independent spatiotemporal coord vbls
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex1),
then sorting 'auxiliary coord vbls'
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex2)
etc... A large amount of deployed met/ocean netCDF is covered by a small
subset of CF.

2

Folks,

Thank you for the references below. I am going to take a look at them.

v/r,
Efren

-----Original Message-----
From: Woolf, A (Andrew) [mailto:A.Woolf@anonymised.com]
Sent: Thursday, December 01, 2005 12:32 AM
To: Rob Atkinson; efren.serra.ctr@anonymised.com
Cc: Alexander Petkov; Norman Barker;
geoserver-devel@lists.sourceforge.net; Dietmar Muller; Keiran Millard;
geotools-devel; Amit Parashar
Subject: RE: [Geoserver-devel] Re: [Geoserver-users] netcdf - wcs
resource

Hi Rob,

am currently trying to organise some effort to get a major
new geotools enhancement (Complex features) into the
Geoserver head, and the WCS support is an obvious target to
try to roll in. Would be grateful if you can identify any
issues, futher development work required to roll that in, and
also if any committed effort for this can be rolled into my
project plan as in-kind support. This sort of in-kind support
will help make the case for the bits of cash we are chasing
to support UI upgrades for the new stuff etc.

Usual story - really want this, but no effort to put in... :frowning:

Is there a UI config for the new WCS stuff? If not, what
needs to be done?

If you stick with a small subset (perhaps one) of netCDF conventions you
can imagine a netCDF collection 'scanner' that picks out e.g.
spatiotemporal metadata defined by those conventions across the fileset,
and presents additional configurable options in a GUI. We're working on
this in NERC DataGrid for CSML - rather than geoserver config -
generation.

In general - we have two requirements in the medium term:
1) auto-configure WCS for a catalogue of NetCDF files, to
support subsetting actions easily having found a netCDF file
in a catalogue

In most atmos/ocean use cases you don't want file-level discovery. You
want the aggregated (timeseries) dataset.

2) be able to subset (in a single WCS call) from collections
of netCDF files across both spatial and temporal dimensions.

Yep.

Also, there are many netCDF conventions - what is the
expected level of support and configurability?

The 'CF' conventions for atmos/ocean data are the canonical standard:
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html
though (as far as I'm aware) not implemented in full *anywhere*.
However, an incremental approach can be taken - e.g. starting with
independent spatiotemporal coord vbls
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex1),
then sorting 'auxiliary coord vbls'
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex2)
etc... A large amount of deployed met/ocean netCDF is covered by a small
subset of CF.

3

HI List, Hy guys,
I am jumping in the conversion quite late, but I was stuck on another
project, sorry.

I see a lot of interest around netCDF and NdCoverages, with plenty of
good ideas to share, therefore my suggestion is, as we already did, to
arrange an IRC session to dig a bit in each others needs, to share
each others' point of view and to lay down a some steps to perform.
If we share the load we could come up with something really good.

A branch both in geotools and in geoserver has been created for the
coverages experiment, next week I should commit some new code to
geotools since I put the old plugins in a much better fit.

At my centre too, they are really interested in netCDF support,
therefore I will spend for sure quite some time next year on
supporting them. I would not mind sharing the work with someone else
with similar requirements. We already met with Bryce Nordgren but if
someone else is willing to join the effort is more than welcome.

I hope to hear good news soon.

Simone.

On 12/1/05, Efren Serra <efren.serra.ctr@anonymised.com> wrote:

Folks,

Thank you for the references below. I am going to take a look at them.

v/r,
Efren

-----Original Message-----
From: Woolf, A (Andrew) [mailto:A.Woolf@anonymised.com]
Sent: Thursday, December 01, 2005 12:32 AM
To: Rob Atkinson; efren.serra.ctr@anonymised.com
Cc: Alexander Petkov; Norman Barker;
geoserver-devel@lists.sourceforge.net; Dietmar Muller; Keiran Millard;
geotools-devel; Amit Parashar
Subject: RE: [Geoserver-devel] Re: [Geoserver-users] netcdf - wcs
resource

Hi Rob,

> am currently trying to organise some effort to get a major
> new geotools enhancement (Complex features) into the
> Geoserver head, and the WCS support is an obvious target to
> try to roll in. Would be grateful if you can identify any
> issues, futher development work required to roll that in, and
> also if any committed effort for this can be rolled into my
> project plan as in-kind support. This sort of in-kind support
> will help make the case for the bits of cash we are chasing
> to support UI upgrades for the new stuff etc.

Usual story - really want this, but no effort to put in... :frowning:

> Is there a UI config for the new WCS stuff? If not, what
> needs to be done?

If you stick with a small subset (perhaps one) of netCDF conventions you
can imagine a netCDF collection 'scanner' that picks out e.g.
spatiotemporal metadata defined by those conventions across the fileset,
and presents additional configurable options in a GUI. We're working on
this in NERC DataGrid for CSML - rather than geoserver config -
generation.

> In general - we have two requirements in the medium term:
> 1) auto-configure WCS for a catalogue of NetCDF files, to
> support subsetting actions easily having found a netCDF file
> in a catalogue

In most atmos/ocean use cases you don't want file-level discovery. You
want the aggregated (timeseries) dataset.

> 2) be able to subset (in a single WCS call) from collections
> of netCDF files across both spatial and temporal dimensions.

Yep.

> Also, there are many netCDF conventions - what is the
> expected level of support and configurability?

The 'CF' conventions for atmos/ocean data are the canonical standard:
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html
though (as far as I'm aware) not implemented in full *anywhere*.
However, an incremental approach can be taken - e.g. starting with
independent spatiotemporal coord vbls
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex1),
then sorting 'auxiliary coord vbls'
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex2)
etc... A large amount of deployed met/ocean netCDF is covered by a small
subset of CF.

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

4

Rob,

Thank you for your response. This is our current state of affairs, where
MBL (Metoc Broker Language), JMBL (Joint Metoc Broker Language):

             +-> MBL <-+ +-> ...
             | | |
Client(s) <--+-> JMBL <-+--> File Encoding Cache <--+-> IEEE format
w/header data source
             | | (GRIB, netCDF, ...) |
             +-> GML <-+ +-> RDBMS with GIS
support (Orcale Spatial, PostgreSQL + PostGIS, ...)

I was wondering, when providing netCDF support for WCS, whether the RDBMS
with GIS support would store metadata pertinent to the netCDF file or
whether it would have to duplicate the existing data in the netCDF file in
the RDBMS with GIS support. Thank you.

v/r,
Efren

-----Original Message-----
From: geoserver-devel-admin@lists.sourceforge.net
[mailto:geoserver-devel-admin@lists.sourceforge.net]On Behalf Of Woolf,
A (Andrew)
Sent: Thursday, December 01, 2005 12:32 AM
To: Rob Atkinson; efren.serra.ctr@anonymised.com
Cc: Alexander Petkov; Norman Barker;
geoserver-devel@lists.sourceforge.net; Dietmar Muller; Keiran Millard;
geotools-devel; Amit Parashar
Subject: RE: [Geoserver-devel] Re: [Geoserver-users] netcdf - wcs
resource

Hi Rob,

am currently trying to organise some effort to get a major
new geotools enhancement (Complex features) into the
Geoserver head, and the WCS support is an obvious target to
try to roll in. Would be grateful if you can identify any
issues, futher development work required to roll that in, and
also if any committed effort for this can be rolled into my
project plan as in-kind support. This sort of in-kind support
will help make the case for the bits of cash we are chasing
to support UI upgrades for the new stuff etc.

Usual story - really want this, but no effort to put in... :frowning:

Is there a UI config for the new WCS stuff? If not, what
needs to be done?

If you stick with a small subset (perhaps one) of netCDF conventions you
can imagine a netCDF collection 'scanner' that picks out e.g.
spatiotemporal metadata defined by those conventions across the fileset,
and presents additional configurable options in a GUI. We're working on
this in NERC DataGrid for CSML - rather than geoserver config -
generation.

In general - we have two requirements in the medium term:
1) auto-configure WCS for a catalogue of NetCDF files, to
support subsetting actions easily having found a netCDF file
in a catalogue

In most atmos/ocean use cases you don't want file-level discovery. You
want the aggregated (timeseries) dataset.

2) be able to subset (in a single WCS call) from collections
of netCDF files across both spatial and temporal dimensions.

Yep.

Also, there are many netCDF conventions - what is the
expected level of support and configurability?

The 'CF' conventions for atmos/ocean data are the canonical standard:
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html
though (as far as I'm aware) not implemented in full *anywhere*.
However, an incremental approach can be taken - e.g. starting with
independent spatiotemporal coord vbls
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex1),
then sorting 'auxiliary coord vbls'
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex2)
etc... A large amount of deployed met/ocean netCDF is covered by a small
subset of CF.

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id865&op=ick
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

5

Efren Serra wrote:

Rob,

Thank you for your response. This is our current state of affairs, where
MBL (Metoc Broker Language), JMBL (Joint Metoc Broker Language):

             +-> MBL <-+ +-> ...
             | | |
Client(s) <--+-> JMBL <-+--> File Encoding Cache <--+-> IEEE format
w/header data source
             | | (GRIB, netCDF, ...) |
             +-> GML <-+ +-> RDBMS with GIS
support (Orcale Spatial, PostgreSQL + PostGIS, ...)

I was wondering, when providing netCDF support for WCS, whether the RDBMS
with GIS support would store metadata pertinent to the netCDF file or
whether it would have to duplicate the existing data in the netCDF file in
the RDBMS with GIS support. Thank you.
  

My money would be on duplication, the RDBMS would want to completly store the Coverage would it not?
(Your arrow goes both ways in your diagram).

That said you may have more luck keeping the searchable metadata in your RDBMS, and running a file cache
on the side to store the actual rasters. If you hide it behind an API it would be identical in use to storing the
coverages in the RDBMS, and would be much less risk then taking on an RnD task.

Question for coverage types, I was working on the GeoTools RnD Page the other day, updating its status & timelines.
Are you still content with the targets (Early next Year = January right now), and descriptions there?

Jody

6

Jody,

Rob,

Thank you for your response. This is our current state of affairs, where
MBL (Metoc Broker Language), JMBL (Joint Metoc Broker Language):

             +-> MBL <-+ +-> ...
             | | |
Client(s) <--+-> JMBL <-+--> File Encoding Cache <--+-> IEEE format
w/header data source
             | | (GRIB, netCDF, ...) |
             +-> GML <-+ +-> RDBMS with

GIS

support (Orcale Spatial, PostgreSQL + PostGIS, ...)

I was wondering, when providing netCDF support for WCS, whether the RDBMS
with GIS support would store metadata pertinent to the netCDF file or
whether it would have to duplicate the existing data in the netCDF file in
the RDBMS with GIS support. Thank you.

My money would be on duplication, the RDBMS would want to completly
store the Coverage would it not?
(Your arrow goes both ways in your diagram).

That said you may have more luck keeping the searchable metadata in your
RDBMS, and running a file cache
on the side to store the actual rasters. If you hide it behind an API it
would be identical in use to storing the
coverages in the RDBMS, and would be much less risk then taking on an
RnD task.

You are correct; I was thinking of an actual file/product cache. The file
encoding cache would
be a dispatcher type application/server.

Question for coverage types, I was working on the GeoTools RnD Page the
other day, updating its status & timelines.
Are you still content with the targets (Early next Year = January right
now), and descriptions there?

Jody

v/r,
Efren

7

If you stick with a small subset (perhaps one) of netCDF conventions you
can imagine a netCDF collection 'scanner' that picks out e.g.
spatiotemporal metadata defined by those conventions across the fileset,
and presents additional configurable options in a GUI. We're working on
this in NERC DataGrid for CSML - rather than geoserver config -
generation.

> In general - we have two requirements in the medium term:
> 1) auto-configure WCS for a catalogue of NetCDF files, to
> support subsetting actions easily having found a netCDF file
> in a catalogue

In most atmos/ocean use cases you don't want file-level discovery. You
want the aggregated (timeseries) dataset.

> 2) be able to subset (in a single WCS call) from collections
> of netCDF files across both spatial and temporal dimensions.

Yep.

> Also, there are many netCDF conventions - what is the
> expected level of support and configurability?

The 'CF' conventions for atmos/ocean data are the canonical standard:
http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html
though (as far as I'm aware) not implemented in full *anywhere*.
However, an incremental approach can be taken - e.g. starting with
independent spatiotemporal coord vbls
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex1),
then sorting 'auxiliary coord vbls'
(http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html#grid_ex2)
etc... A large amount of deployed met/ocean netCDF is covered by a small
subset of CF.

The netCDF java library already supports some netCDF conventions. This
is the list from the package:

ADASConvention.java
ATDRadarConvention.java
AWIPSConvention.java
AWIPSsatConvention.java
CF1Convention.java
COARDSConvention.java
CSMConvention.java
GDVConvention.java
GIEFConvention.java
GribConvention.java
IFPSConvention.java
M3IOConvention.java
MADISStation.java
NUWGConvention.java
UnidataObsConvention.java
WRFConvention.java
ZebraConvention.java