[Geoserver-devel] GSIP 158 - NetCDF Output Support for Variable Attributes and Extra Variables

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158

[...]

Motivation

NetCDF/GRIB sources contain rich metadata in the form of variable attributes and extra variables that give meaning to data. When a coverage is read from a NetCDF/GRIB source, this information is lost, even if the output from GeoServer is NetCDF that is capable of representing this information.

Proposal

Functionality

For NetCDF output add:

1. An option to allow all attributes to be copied from the source NetCDF/GRIB variable to the target variable

2. Support for manual configuration of variable attributes, much like the current support for setting global attributes

3. Support for configuration of extra variables which are copied from the NetCDF/GRIB source to the output; initially only scalar variables will be supported

4. An option for each extra variable to be expanded over dimensions, that is, values copied from one scalar per ImageMosaic granule are assembled into a multidimensional variable over, for example, time and elevation

[...]

The GSIP has screenshots of the current settings tab and a rough draft of the proposed settings tab.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

I have added some GitHub branch comparisons in the implementation section. The implementation is incomplete; discussion comes first. It is still too early for pull requests.

On 25/03/17 16:59, Ben Caradoc-Davies wrote:

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

I have submitted pull requests for review.

See the GSIP 158 page for screenshot and user manual:
https://github.com/geoserver/geoserver/wiki/GSIP-158

A small change is required to GeoTools:

[GEOT-5709] Support coverage property SourceUrl
https://github.com/geotools/geotools/pull/1564
https://osgeo-org.atlassian.net/browse/GEOT-5709

Most of the implementation is in GeoServer:

[GEOS-8099] GSIP 158 - NetCDF output support for variable attributes and extra variables
https://github.com/geoserver/geoserver/pull/2281
https://osgeo-org.atlassian.net/browse/GEOS-8099

Kind regards,
Ben.

On 25/03/17 17:15, Ben Caradoc-Davies wrote:

I have added some GitHub branch comparisons in the implementation
section. The implementation is incomplete; discussion comes first. It is
still too early for pull requests.

On 25/03/17 16:59, Ben Caradoc-Davies wrote:

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hi Ben,
I’ve lost track of this one, seems like a step was skipped?
No big deal, but checking the history I see GSIP-158 switching from
a “preliminary discussion” thread (unfortunately empty) to a pull
request without a vote in the middle.

Have I missed it?

No big deal, we can vote and review at the same time.

Cheers
Andrea

···

On Tue, Apr 25, 2017 at 5:09 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

I have submitted pull requests for review.

See the GSIP 158 page for screenshot and user manual:
https://github.com/geoserver/geoserver/wiki/GSIP-158

A small change is required to GeoTools:

[GEOT-5709] Support coverage property SourceUrl
https://github.com/geotools/geotools/pull/1564
https://osgeo-org.atlassian.net/browse/GEOT-5709

Most of the implementation is in GeoServer:

[GEOS-8099] GSIP 158 - NetCDF output support for variable attributes and
extra variables
https://github.com/geoserver/geoserver/pull/2281
https://osgeo-org.atlassian.net/browse/GEOS-8099

Kind regards,
Ben.

On 25/03/17 17:15, Ben Caradoc-Davies wrote:

I have added some GitHub branch comparisons in the implementation
section. The implementation is incomplete; discussion comes first. It is
still too early for pull requests.

On 25/03/17 16:59, Ben Caradoc-Davies wrote:

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Andrea,

this proposal is (almost) entirely confined to one module (gs-netcdf-out), changes no API, is (almost) fully backwards compatible in behaviour, and is fully backwards compatible in serialised settings. The GSIP might not even be required, but I thought it a useful way to ask for comments. I do not expect it to be contentious; I welcome all feedback.

There was some prior discussion on GeoTools PR #1417 <https://github.com/geotools/geotools/pull/1417&gt; which was most helpful in dissuading me from that approach, and I also got an off-list analysis of the cost of opening NetCDF/GRIB files (I have asked for permission to copy it as a Jira comment).

I agree that there was no response to my original proposal, perhaps because of the REST migration sprint, and perhaps because a module maintainer was on leave. The only feedback I got on this one was your comment on GEOT-5709 (thanks!). I hope that the screenshot, user manual section, and fully implemented pull requests will attract more attention.

Kind regards,
Ben.

On 25/04/17 18:44, Andrea Aime wrote:

Hi Ben,
I've lost track of this one, seems like a step was skipped?
No big deal, but checking the history I see GSIP-158 switching from
a "preliminary discussion" thread (unfortunately empty) to a pull
request without a vote in the middle.

Have I missed it?

No big deal, we can vote and review at the same time.

Cheers
Andrea

On Tue, Apr 25, 2017 at 5:09 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I have submitted pull requests for review.

See the GSIP 158 page for screenshot and user manual:
https://github.com/geoserver/geoserver/wiki/GSIP-158

A small change is required to GeoTools:

[GEOT-5709] Support coverage property SourceUrl
https://github.com/geotools/geotools/pull/1564
https://osgeo-org.atlassian.net/browse/GEOT-5709

Most of the implementation is in GeoServer:

[GEOS-8099] GSIP 158 - NetCDF output support for variable attributes and
extra variables
https://github.com/geoserver/geoserver/pull/2281
https://osgeo-org.atlassian.net/browse/GEOS-8099

Kind regards,
Ben.

On 25/03/17 17:15, Ben Caradoc-Davies wrote:

I have added some GitHub branch comparisons in the implementation
section. The implementation is incomplete; discussion comes first. It is
still too early for pull requests.

On 25/03/17 16:59, Ben Caradoc-Davies wrote:

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

On Tue, Apr 25, 2017 at 1:59 PM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

Andrea,

this proposal is (almost) entirely confined to one module (gs-netcdf-out),
changes no API, is (almost) fully backwards compatible in behaviour, and is
fully backwards compatible in serialised settings. The GSIP might not even
be required, but I thought it a useful way to ask for comments. I do not
expect it to be contentious; I welcome all feedback.

Hi Ben,
I did not look at it and don't know if it's proposal worthy. But I believe
that, message wise, it's problematic to stand up a
proposal and then move on without a discussion on it, others might follow
in your footsteps for stuff that is truly
proposal worthy (because it happened before, you know, it sets a precedent,
and people are not always inclined
to check the details or context and just get the narrative suiting them and
go with it).

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

You could move the page to https://github.com/geoserver/geoserver/wiki/Maintenance if a vote is not needed.

I would also be willing to combine the proposal page and the maintenance page so we get a consistent breakdown of changes by version number.

···

On 25 April 2017 at 04:59, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Andrea,

this proposal is (almost) entirely confined to one module
(gs-netcdf-out), changes no API, is (almost) fully backwards compatible
in behaviour, and is fully backwards compatible in serialised settings.
The GSIP might not even be required, but I thought it a useful way to
ask for comments. I do not expect it to be contentious; I welcome all
feedback.

There was some prior discussion on GeoTools PR #1417
<https://github.com/geotools/geotools/pull/1417> which was most helpful
in dissuading me from that approach, and I also got an off-list analysis
of the cost of opening NetCDF/GRIB files (I have asked for permission to
copy it as a Jira comment).

I agree that there was no response to my original proposal, perhaps
because of the REST migration sprint, and perhaps because a module
maintainer was on leave. The only feedback I got on this one was your
comment on GEOT-5709 (thanks!). I hope that the screenshot, user manual
section, and fully implemented pull requests will attract more attention.

Kind regards,
Ben.

On 25/04/17 18:44, Andrea Aime wrote:

Hi Ben,
I’ve lost track of this one, seems like a step was skipped?
No big deal, but checking the history I see GSIP-158 switching from
a “preliminary discussion” thread (unfortunately empty) to a pull
request without a vote in the middle.

Have I missed it?

No big deal, we can vote and review at the same time.

Cheers
Andrea

On Tue, Apr 25, 2017 at 5:09 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I have submitted pull requests for review.

See the GSIP 158 page for screenshot and user manual:
https://github.com/geoserver/geoserver/wiki/GSIP-158

A small change is required to GeoTools:

[GEOT-5709] Support coverage property SourceUrl
https://github.com/geotools/geotools/pull/1564
https://osgeo-org.atlassian.net/browse/GEOT-5709

Most of the implementation is in GeoServer:

[GEOS-8099] GSIP 158 - NetCDF output support for variable attributes and
extra variables
https://github.com/geoserver/geoserver/pull/2281
https://osgeo-org.atlassian.net/browse/GEOS-8099

Kind regards,
Ben.

On 25/03/17 17:15, Ben Caradoc-Davies wrote:

I have added some GitHub branch comparisons in the implementation
section. The implementation is incomplete; discussion comes first. It is
still too early for pull requests.

On 25/03/17 16:59, Ben Caradoc-Davies wrote:

Preliminary GSIP for discussion, no Jira issues yet:
https://github.com/geoserver/geoserver/wiki/GSIP-158


Ben Caradoc-Davies <ben@anonymised.com91…>
Director
Transient Software Limited <http://transient.nz/>
New Zealand



Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Jody Garnett

On 26/04/17 00:08, Andrea Aime wrote:

I did not look at it and don't know if it's proposal worthy. But I believe
that, message wise, it's problematic to stand up a
proposal and then move on without a discussion on it, others might follow
in your footsteps for stuff that is truly
proposal worthy (because it happened before, you know, it sets a precedent,
and people are not always inclined
to check the details or context and just get the narrative suiting them and
go with it).

Andrea, I agree. It is also risky to implement a proposal when someone might have a good objection that requires a completely different approach and much rework. The purpose of a GSIP is to obtain agreement before expending effort on an implementation. I did not know how to proceed in the absence of any response to the discussion thread or to other list emails to requesters of these features. Would it have been better to call for a vote to force discussion? I was hoping that a concrete implementation would attract more attention. In any case, I will wait for comments and reviews and change the implementation if required.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

On Tue, Apr 25, 2017 at 11:01 PM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

Andrea, I agree. It is also risky to implement a proposal when someone
might have a good objection that requires a completely different approach
and much rework. The purpose of a GSIP is to obtain agreement before
expending effort on an implementation. I did not know how to proceed in the
absence of any response to the discussion thread or to other list emails to
requesters of these features. Would it have been better to call for a vote
to force discussion?

Hum... I am not sure, but something like this might have worked:

   - "Threaten" to go to vote on the original thread as a follow up (it
   seems nobody has a problem with this, so I'll start the vote in 1-2 days)
   - Vote for good if no more feedback comes on

I was hoping that a concrete implementation would attract more attention.
In any case, I will wait for comments and reviews and change the
implementation if required.

Yep, implementing along before vote is not prohibited, it just increases
the risk of reworking and it's all on you :-p

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

On 26/04/17 19:35, Andrea Aime wrote:

Hum... I am not sure, but something like this might have worked:
   - "Threaten" to go to vote on the original thread as a follow up (it
   seems nobody has a problem with this, so I'll start the vote in 1-2 days)
   - Vote for good if no more feedback comes on

That sounds good to me.

Yep, implementing along before vote is not prohibited, it just increases
the risk of reworking and it's all on you :-p

Indeed it is, but feedback is valuable and I will not begrudge some rework if it makes the final product better. :slight_smile:

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand