[Geoserver-devel] A note on build failures due to url mangler changes (url encoding and decoding)

Hi,
the build failure you've seen is due to the URL manglers GSIP
and me not checking the app schema module.

The issue there is that the new ResponseUtils.buildURL method
does actually url-encode all kvp parameters.
And this is a good thing, right?

Well, yeah, but it turns out we used to put in the various
URL we generated characters like the comma or the column that
are actually reserved chars which are not supposed to be found
in a URL, see here:
http://www.blooberry.com/indexdot/html/topics/urlencoding.htm
http://www.permadi.com/tutorial/urlEncoding/
http://en.wikipedia.org/wiki/Percent-encoding

and there is an online url encoder, decoder:
http://www.albionresearch.com/misc/urlencode.php

So, if I'm not reading the above documents wrong, the
current behavior is correct and some tests needed to
be fixed and url-decode the urls contained in our xml documents
before trying to split them on "," or checking equality with strings containing a column.

I'm still a bit surprised about this finding but...
here it goes.

If you know more about this topic and care to share it I'll be all ears

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Andrea Aime wrote:

Hi,
the build failure you've seen is due to the URL manglers GSIP
and me not checking the app schema module.

The issue there is that the new ResponseUtils.buildURL method
does actually url-encode all kvp parameters.
And this is a good thing, right?

Well, yeah, but it turns out we used to put in the various
URL we generated characters like the comma or the column that
are actually reserved chars which are not supposed to be found
in a URL, see here:
http://www.blooberry.com/indexdot/html/topics/urlencoding.htm
http://www.permadi.com/tutorial/urlEncoding/
http://en.wikipedia.org/wiki/Percent-encoding

and there is an online url encoder, decoder:
http://www.albionresearch.com/misc/urlencode.php

So, if I'm not reading the above documents wrong, the
current behavior is correct and some tests needed to
be fixed and url-decode the urls contained in our xml documents
before trying to split them on "," or checking equality with strings containing a column.

I'm still a bit surprised about this finding but...
here it goes.

If you know more about this topic and care to share it I'll be all ears

Any outgoing URL _parameter value_ should be url encoded, that way parameter values can't conflict with reserved characters like '&', '?', '=' etc., so yeah, anyone parsing out an outgoing (a geoserver generated) url with CGI parameters should decode the parameter value. By the other hand, I'm pretty sure when any URL request with CGI parameters comming into GeoServer where the parameter values are urlencoded are automaticalle decoded by the servlet container, so no need to decode those ones (ie, kvp parsers).

I'm not sure where these error was occurring, but it makes sense.
Cheers,
Gabriel

Cheers
Andrea

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Gabriel Roldan ha scritto:

Andrea Aime wrote:

Hi,
the build failure you've seen is due to the URL manglers GSIP
and me not checking the app schema module.

The issue there is that the new ResponseUtils.buildURL method
does actually url-encode all kvp parameters.
And this is a good thing, right?

Well, yeah, but it turns out we used to put in the various
URL we generated characters like the comma or the column that
are actually reserved chars which are not supposed to be found
in a URL, see here:
http://www.blooberry.com/indexdot/html/topics/urlencoding.htm
http://www.permadi.com/tutorial/urlEncoding/
http://en.wikipedia.org/wiki/Percent-encoding

and there is an online url encoder, decoder:
http://www.albionresearch.com/misc/urlencode.php

So, if I'm not reading the above documents wrong, the
current behavior is correct and some tests needed to
be fixed and url-decode the urls contained in our xml documents
before trying to split them on "," or checking equality with strings containing a column.

I'm still a bit surprised about this finding but...
here it goes.

If you know more about this topic and care to share it I'll be all ears

Any outgoing URL _parameter value_ should be url encoded, that way parameter values can't conflict with reserved characters like '&', '?', '=' etc., so yeah, anyone parsing out an outgoing (a geoserver generated) url with CGI parameters should decode the parameter value. By the other hand, I'm pretty sure when any URL request with CGI parameters comming into GeoServer where the parameter values are urlencoded are automaticalle decoded by the servlet container, so no need to decode those ones (ie, kvp parsers).

I'm not sure where these error was occurring, but it makes sense.

Well, basically most of the DescribeFeatureType back links were
not url-encoding the column and the commas in the parameter value,
e.g.:
typeName=topp:states,sf:roads
instead of the url-encoded version:
topp%3Astates%2Csf%3Aroads

And then unit tests were freely grabbing stuff from the xml
document and splitting over "," without doing any url-decoding.

One thing I know for sure is that the ( and ) are not required
to be encoded, but the Java URLEncoder does it anyways (however,
it does not seem to be harmful).

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.