[Geoserver-devel] Reverted fix to get release going, GetCapabilties handling headaches

Hi,
before doing the 2.6.2 release I’ve checked the CITE tests status and
found WCS 1.1 tests have been failing on the stable series since
December 23rd

Found the fixes for this issue caused the regression, and reverted them:
http://jira.codehaus.org/browse/GEOS-6807

Now… this pushes a bug back in, but we cannot release non compliant
code on the stable series.
Also, looking into it, I’ve found it to be a sizeable can of worms.

The issues here is in KVP parsing, Nicola’s changes fixed a problem
where the code was parsing a Sections object of the wrong type
for the GetCapabilitiesType object being built, basically, we were
building one compatible with WCS 1.1 instead of WCS 2.0.

WCS is particularly nasty in this respect, as it has 3 different object
models (the three specs being so different), and each one is based
on a different OWS version, which means we have around three
SectionsType objects.

The issue with Nicola’s fix is that he made the kvp parsers version
bound, but when a GetCapabiltiies request is made, the version
parameter is normally missing, so none of the KVP parsers will
be picked. And the CITE tests make a sizeable number of these
calls, either with nothing, or with acceptVersions (or with accept_versions
too… which is a bug in the CITE tests themselves).

So… how do we get out of this? I believe the Dispatcher should
be modified so that the version negotiation is done before any
KVP parsing is carried out.
First, it should determine the service (we have heuristics here, like using the
last bit of the path if service=xyz is missing), and then determine the request.

Once we know what service and request, we can reason on version.
If the request is not a GetCapabilites, we look at version, if missing we
adopt the usual heuristics of picking the highest for the chosen service,
and only then we do the KVP parsing, at this point we have a firm
service and version so we can lookup the service/version bound
kvp parsers correctly.

If the request is GetCapabilites instead, things get more fun.
We first check if acceptVersions (or accept_versions) has been provided,
if so, we match it with the available versions for the service provided,
and do the negotiation.
If that is missing, we check for version, which should not be part of a GetCapabilities,
but we are used to handle it, so we should keep on doing that… and also
check for wmtver, which is an old WMS specific way to set the version.

At this point we should again know firmly which version has been matched,
and thus we can do the KVP parsing correclty.

Sounds like a plan?

There is of course one downside, this is a significant change to the Dispatcher,
if I get some feedback early today I can try to code the changes by the
evening and at least include them in the beta, but for 2.6.2 I believe
it’s too risky

Opinions?

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Agree with the plan, and since it relates to conformance I feel it is worth getting it done for the beta.

I may be able to delay the beta one day if needed, and we can back port to stable once the beta is out.

···

On 18 January 2015 at 02:11, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
before doing the 2.6.2 release I’ve checked the CITE tests status and
found WCS 1.1 tests have been failing on the stable series since
December 23rd

Found the fixes for this issue caused the regression, and reverted them:
http://jira.codehaus.org/browse/GEOS-6807

Now… this pushes a bug back in, but we cannot release non compliant
code on the stable series.
Also, looking into it, I’ve found it to be a sizeable can of worms.

The issues here is in KVP parsing, Nicola’s changes fixed a problem
where the code was parsing a Sections object of the wrong type
for the GetCapabilitiesType object being built, basically, we were
building one compatible with WCS 1.1 instead of WCS 2.0.

WCS is particularly nasty in this respect, as it has 3 different object
models (the three specs being so different), and each one is based
on a different OWS version, which means we have around three
SectionsType objects.

The issue with Nicola’s fix is that he made the kvp parsers version
bound, but when a GetCapabiltiies request is made, the version
parameter is normally missing, so none of the KVP parsers will
be picked. And the CITE tests make a sizeable number of these
calls, either with nothing, or with acceptVersions (or with accept_versions
too… which is a bug in the CITE tests themselves).

So… how do we get out of this? I believe the Dispatcher should
be modified so that the version negotiation is done before any
KVP parsing is carried out.
First, it should determine the service (we have heuristics here, like using the
last bit of the path if service=xyz is missing), and then determine the request.

Once we know what service and request, we can reason on version.
If the request is not a GetCapabilites, we look at version, if missing we
adopt the usual heuristics of picking the highest for the chosen service,
and only then we do the KVP parsing, at this point we have a firm
service and version so we can lookup the service/version bound
kvp parsers correctly.

If the request is GetCapabilites instead, things get more fun.
We first check if acceptVersions (or accept_versions) has been provided,
if so, we match it with the available versions for the service provided,
and do the negotiation.
If that is missing, we check for version, which should not be part of a GetCapabilities,
but we are used to handle it, so we should keep on doing that… and also
check for wmtver, which is an old WMS specific way to set the version.

At this point we should again know firmly which version has been matched,
and thus we can do the KVP parsing correclty.

Sounds like a plan?

There is of course one downside, this is a significant change to the Dispatcher,
if I get some feedback early today I can try to code the changes by the
evening and at least include them in the beta, but for 2.6.2 I believe
it’s too risky

Opinions?

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Jody Garnett

On Sun, Jan 18, 2015 at 5:02 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Agree with the plan, and since it relates to conformance I feel it is
worth getting it done for the beta.

I may be able to delay the beta one day if needed, and we can back port to
stable once the beta is out.

I have mixed news, managed to move the code enough to have kvp parsing run
after service
and version are set (including enough negotiation to figure out the
version), and fixed the build
from failures resulting from that change.

However I did not manage to run the cite tests with that change, nor I know
yet if Nicola's
patches will work with the changes I've made.
But.. Sunday is over. Not sure if anyone wants to take over the work,
I've landed it here:

https://github.com/aaime/geoserver/tree/kvp_resolve
Rebased it to a single commit:
https://github.com/aaime/geoserver/commit/424067c647d8397f8dfbb8da6afe8f5cf69521dd

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

I am not sure how to help on this one Andrea, I am not in position to run cite tests locally (missed out on that part of the foss4g sprint).

···

On 18 January 2015 at 12:04, Andrea Aime <andrea.aime@anonymised.com> wrote:


Jody Garnett

On Sun, Jan 18, 2015 at 5:02 PM, Jody Garnett <jody.garnett@anonymised.com> wrote:

Agree with the plan, and since it relates to conformance I feel it is worth getting it done for the beta.

I may be able to delay the beta one day if needed, and we can back port to stable once the beta is out.

I have mixed news, managed to move the code enough to have kvp parsing run after service
and version are set (including enough negotiation to figure out the version), and fixed the build
from failures resulting from that change.

However I did not manage to run the cite tests with that change, nor I know yet if Nicola’s
patches will work with the changes I’ve made.
But… Sunday is over. Not sure if anyone wants to take over the work,
I’ve landed it here:

https://github.com/aaime/geoserver/tree/kvp_resolve

Rebased it to a single commit:
https://github.com/aaime/geoserver/commit/424067c647d8397f8dfbb8da6afe8f5cf69521dd

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Mon, Jan 19, 2015 at 6:28 AM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

I am not sure how to help on this one Andrea, I am not in position to run
cite tests locally (missed out on that part of the foss4g sprint).

Hi Jody,
we discussed with Nicola another approach to the same bug fix, that does
not require a last minute change to the dispatcher,
and succesfully tested it on trunk against the CITE tests.
I've backported it to 2.6.x and I'm waiting for hudson to go though
2.6.x/nightly/cite cycle, if all goes well tomorrow I should
be good to go with the 2.6.2 release.

About the pull request in this thread, I'll keep it there and give it
proper testing once the feature freeze is over

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------