[Geoserver-devel] REST configuration and proxying

Andrea_Aime3 · September 7, 2009, 4:45pm

Hi,
I'm working on GSIP39 and I've stumbled in some odd
code in the REST and RESTConfig modules, so I'm looking
for some clarifications.

The first bit of code is in RESTDispatcher, line 146:

---------------------------------------------------------

//set the page uri's, proxying if necessary
String rootURI = request.getRootRef().toString();
String baseURI = null;
if ( request.getResourceRef().getBaseRef() != null ) {
baseURI = request.getResourceRef().getBaseRef().toString();
}
String pageURI = request.getResourceRef().toString();

if ( gs.getGlobal().getProxyBaseUrl() != null ) {
     String host =
         RequestUtils.proxifiedBaseURL(request.getHostRef().toString(), gs.getGlobal().getProxyBaseUrl());
     rootURI = ResponseUtils.appendPath( host, request.getRootRef().getPath() );
     baseURI = baseURI != null ?
             ResponseUtils.appendPath( host, request.getResourceRef().getBaseRef().getPath() ) : baseURI;
     pageURI = ResponseUtils.appendPath( host, request.getResourceRef().getPath() );
}

---------------------------------------------------------

Three URI are built, root, base and page, which are
then stored in "PageInfo".
First off, what is the meaning of the three uris?
I don't see it explained anywhere.

Second thing I've noticed is that PageInfo.baseURI is not used
anywhere, so I'm wondering why it is built to start with.

Another thing to notice is that the rootURI is two
different beasts depending on whether proxying is enabled.
If there is no proxing debugging in the tests I get:
rootURI=http://localhost/geoserver/rest
but if proxying is enabled, host ref is used, meaning
rootURI=http://localhost
before passing it down to the proxyfing code...

Unfortunately none of the two is what the proxy code expects,
the baseURL expected by the code is "http://host:port/appname" instead,
something I don't know, in general, how to extract out
of a restlet request.

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request

One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Justin_Deoliveira · September 8, 2009, 3:04am

Andrea Aime wrote:

Hi,
I'm working on GSIP39 and I've stumbled in some odd
code in the REST and RESTConfig modules, so I'm looking
for some clarifications.

The first bit of code is in RESTDispatcher, line 146:

---------------------------------------------------------

//set the page uri's, proxying if necessary
String rootURI = request.getRootRef().toString();
String baseURI = null;
if ( request.getResourceRef().getBaseRef() != null ) {
     baseURI = request.getResourceRef().getBaseRef().toString();
}
String pageURI = request.getResourceRef().toString();

if ( gs.getGlobal().getProxyBaseUrl() != null ) {
     String host =
         RequestUtils.proxifiedBaseURL(request.getHostRef().toString(), gs.getGlobal().getProxyBaseUrl());
     rootURI = ResponseUtils.appendPath( host, request.getRootRef().getPath() );
     baseURI = baseURI != null ?
             ResponseUtils.appendPath( host, request.getResourceRef().getBaseRef().getPath() ) : baseURI;
     pageURI = ResponseUtils.appendPath( host, request.getResourceRef().getPath() );
}

---------------------------------------------------------

Three URI are built, root, base and page, which are
then stored in "PageInfo".
First off, what is the meaning of the three uris?
I don't see it explained anywhere.

The root uri is the the host + "/rest" context. It is used when resource references need to be encoded from the root of a resource hierarchy.

The page uri is the uri of the resource being requested. The base uri is the parent of the page uri.

Example:

page = http://localhost:8080/geoserver/rest/x/y/z.foo
base = http://localhost:8080/geoserver/rest/x/y
root = http://localhost:8080/geoserver/rest

Second thing I've noticed is that PageInfo.baseURI is not used
anywhere, so I'm wondering why it is built to start with.

At one point it was used in the freemarker html templates, but it does not seem to be the case. So yeah, we could probably kill it, i just left it in for completeness.

Another thing to notice is that the rootURI is two
different beasts depending on whether proxying is enabled.
If there is no proxing debugging in the tests I get:
rootURI=http://localhost/geoserver/rest
but if proxying is enabled, host ref is used, meaning
rootURI=http://localhost
before passing it down to the proxyfing code...

Yeah... that does look fishy. Probably just a bug that noone has run into proxying with restconfig.

Unfortunately none of the two is what the proxy code expects,
the baseURL expected by the code is "http://host:port/appname" instead,
something I don't know, in general, how to extract out
of a restlet request.

I don't see a quick way through the api. But there is a method in ResponseUtils that does this:

ResponseUtils.getParentUrl(request.getRootRef().toString())

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request

One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Not sure I quite understand. Can the rest code just run the built url through the mangler extension point before it encodes it?

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Andrea_Aime3 · September 8, 2009, 7:03am

Justin Deoliveira ha scritto:

Andrea Aime wrote:

Hi,
I'm working on GSIP39 and I've stumbled in some odd
code in the REST and RESTConfig modules, so I'm looking
for some clarifications.

The first bit of code is in RESTDispatcher, line 146:

---------------------------------------------------------

//set the page uri's, proxying if necessary
String rootURI = request.getRootRef().toString();
String baseURI = null;
if ( request.getResourceRef().getBaseRef() != null ) {
     baseURI = request.getResourceRef().getBaseRef().toString();
}
String pageURI = request.getResourceRef().toString();

if ( gs.getGlobal().getProxyBaseUrl() != null ) {
     String host =
         RequestUtils.proxifiedBaseURL(request.getHostRef().toString(), gs.getGlobal().getProxyBaseUrl());
     rootURI = ResponseUtils.appendPath( host, request.getRootRef().getPath() );
     baseURI = baseURI != null ?
             ResponseUtils.appendPath( host, request.getResourceRef().getBaseRef().getPath() ) : baseURI;
     pageURI = ResponseUtils.appendPath( host, request.getResourceRef().getPath() );
}

---------------------------------------------------------

Three URI are built, root, base and page, which are
then stored in "PageInfo".
First off, what is the meaning of the three uris?
I don't see it explained anywhere.

The root uri is the the host + "/rest" context. It is used when resource references need to be encoded from the root of a resource hierarchy.

The page uri is the uri of the resource being requested. The base uri is the parent of the page uri.

Example:

page = http://localhost:8080/geoserver/rest/x/y/z.foo
base = http://localhost:8080/geoserver/rest/x/y
root = http://localhost:8080/geoserver/rest

Second thing I've noticed is that PageInfo.baseURI is not used
anywhere, so I'm wondering why it is built to start with.

At one point it was used in the freemarker html templates, but it does not seem to be the case. So yeah, we could probably kill it, i just left it in for completeness.

I have no problems leaving it as long as we know what it is.
Proxying wise the following set would be more useful, as it makes
the baseurl evident and allows the code to still build piecemeal
the path portion

baseurl = http://localhost:8080/geoserver/
rootpath = rest
basepath = rest/x/y
pagepath = rest/x/y/z.foo

Another thing to notice is that the rootURI is two
different beasts depending on whether proxying is enabled.
If there is no proxing debugging in the tests I get:
rootURI=http://localhost/geoserver/rest
but if proxying is enabled, host ref is used, meaning
rootURI=http://localhost
before passing it down to the proxyfing code...

Yeah... that does look fishy. Probably just a bug that noone has run into proxying with restconfig.

Unfortunately none of the two is what the proxy code expects,
the baseURL expected by the code is "http://host:port/appname" instead,
something I don't know, in general, how to extract out
of a restlet request.

I don't see a quick way through the api. But there is a method in ResponseUtils that does this:

ResponseUtils.getParentUrl(request.getRootRef().toString())

Sounds good.

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request

One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Not sure I quite understand. Can the rest code just run the built url through the mangler extension point before it encodes it?

Yes and no. At the moment the code proxifies the URI
before building the PageInfo, and then appends stuff (formats)
to those proxified URIs in another class.
What needs doing is storing the original URI in the PageInfo
instead, and have the code building the final URL use
the originals, extract the expected base URL (or just have it,
like in the proposal above), add whatever else is needed to the path
portion and finally call ResponseUtil.buildURL

However that would make PageInfo something that is not a good
idea to pass over to the templates (as it would contain
un-proxied stuff). Suggestions?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Justin_Deoliveira · September 8, 2009, 2:56pm

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Andrea Aime wrote:

Hi,
I'm working on GSIP39 and I've stumbled in some odd
code in the REST and RESTConfig modules, so I'm looking
for some clarifications.

The first bit of code is in RESTDispatcher, line 146:

---------------------------------------------------------

//set the page uri's, proxying if necessary
String rootURI = request.getRootRef().toString();
String baseURI = null;
if ( request.getResourceRef().getBaseRef() != null ) {
     baseURI = request.getResourceRef().getBaseRef().toString();
}
String pageURI = request.getResourceRef().toString();

if ( gs.getGlobal().getProxyBaseUrl() != null ) {
     String host =
         RequestUtils.proxifiedBaseURL(request.getHostRef().toString(), gs.getGlobal().getProxyBaseUrl());
     rootURI = ResponseUtils.appendPath( host, request.getRootRef().getPath() );
     baseURI = baseURI != null ?
             ResponseUtils.appendPath( host, request.getResourceRef().getBaseRef().getPath() ) : baseURI;
     pageURI = ResponseUtils.appendPath( host, request.getResourceRef().getPath() );
}

---------------------------------------------------------

Three URI are built, root, base and page, which are
then stored in "PageInfo".
First off, what is the meaning of the three uris?
I don't see it explained anywhere.

The root uri is the the host + "/rest" context. It is used when resource references need to be encoded from the root of a resource hierarchy.

The page uri is the uri of the resource being requested. The base uri is the parent of the page uri.

Example:

page = http://localhost:8080/geoserver/rest/x/y/z.foo
base = http://localhost:8080/geoserver/rest/x/y
root = http://localhost:8080/geoserver/rest

Second thing I've noticed is that PageInfo.baseURI is not used
anywhere, so I'm wondering why it is built to start with.

At one point it was used in the freemarker html templates, but it does not seem to be the case. So yeah, we could probably kill it, i just left it in for completeness.

I have no problems leaving it as long as we know what it is.
Proxying wise the following set would be more useful, as it makes
the baseurl evident and allows the code to still build piecemeal
the path portion

baseurl = http://localhost:8080/geoserver/
rootpath = rest
basepath = rest/x/y
pagepath = rest/x/y/z.foo

Sounds good.

Another thing to notice is that the rootURI is two
different beasts depending on whether proxying is enabled.
If there is no proxing debugging in the tests I get:
rootURI=http://localhost/geoserver/rest
but if proxying is enabled, host ref is used, meaning
rootURI=http://localhost
before passing it down to the proxyfing code...

Yeah... that does look fishy. Probably just a bug that noone has run into proxying with restconfig.

Unfortunately none of the two is what the proxy code expects,
the baseURL expected by the code is "http://host:port/appname" instead,
something I don't know, in general, how to extract out
of a restlet request.

I don't see a quick way through the api. But there is a method in ResponseUtils that does this:

ResponseUtils.getParentUrl(request.getRootRef().toString())

Sounds good.

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request

One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Not sure I quite understand. Can the rest code just run the built url through the mangler extension point before it encodes it?

Yes and no. At the moment the code proxifies the URI
before building the PageInfo, and then appends stuff (formats)
to those proxified URIs in another class.
What needs doing is storing the original URI in the PageInfo
instead, and have the code building the final URL use
the originals, extract the expected base URL (or just have it,
like in the proposal above), add whatever else is needed to the path
portion and finally call ResponseUtil.buildURL

However that would make PageInfo something that is not a good
idea to pass over to the templates (as it would contain
un-proxied stuff). Suggestions?

So, just to ensure I understand the issue. The issue is that the rest dispatcher will first try to mangle the url, so in the case of proxying and the case (which I think is TIKE) append some kvp parameters. So taken the url:

http://host/geoserver/rest/foo.bar

It would mangle to:

http://proxyhost/geoserver/rest/foo.bar?key=value

Now... the problem is that when restconfig gets a hold of this to do resource liking it will turn it to:

http://proxyhost/geoserver/rest/foo/child.bar

And loose the kvp previously added?

One possible solution is to have the restconfig just call buildURL again. I guess to have this work it might be a good idea to have the following always hold true:

buildURL(original) = buildURL(buildURL(original))

However there is still the issue of the tempaltes which require PageInfo to get the urls they need to spit out. One possible solution would be to give access to give the freemarker templates access to the buildURL function in some shape or form. Perhaps just adding a global function or just making it a function on page info? Something like:

${page.uri(page.pageURI)}

There PageInfo#uri would just be a call through to ResponseUtils.buildURL()

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Andrea_Aime3 · September 8, 2009, 3:12pm

Justin Deoliveira ha scritto:

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request
One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Not sure I quite understand. Can the rest code just run the built url through the mangler extension point before it encodes it?

Yes and no. At the moment the code proxifies the URI
before building the PageInfo, and then appends stuff (formats)
to those proxified URIs in another class.
What needs doing is storing the original URI in the PageInfo
instead, and have the code building the final URL use
the originals, extract the expected base URL (or just have it,
like in the proposal above), add whatever else is needed to the path
portion and finally call ResponseUtil.buildURL

However that would make PageInfo something that is not a good
idea to pass over to the templates (as it would contain
un-proxied stuff). Suggestions?

So, just to ensure I understand the issue. The issue is that the rest dispatcher will first try to mangle the url, so in the case of proxying and the case (which I think is TIKE) append some kvp parameters. So taken the url:

http://host/geoserver/rest/foo.bar

It would mangle to:

http://proxyhost/geoserver/rest/foo.bar?key=value

Now... the problem is that when restconfig gets a hold of this to do resource liking it will turn it to:

http://proxyhost/geoserver/rest/foo/child.bar

Worse, it would end up just doing and appendPath which will result in:

http://proxyhost/geoserver/rest/foo.bar?key=value/child.bar

One possible solution is to have the restconfig just call buildURL again. I guess to have this work it might be a good idea to have the following always hold true:

buildURL(original) = buildURL(buildURL(original))

Mind buildURL takes quite a bit of parameters:

buildURL(String baseURL, String path, Map<String, String> kvp, URLType type)

As for having calls twice in a row be equivalent, that is not possible
as the kvp is embedded in the resulting path.
We'd need some way to re-parse is back separating the base url,
the path, and the kvp. As for having it work so that applying it
twice would lead to the same results we'd have to ask the
manglers to respect that rule.
But as I said, that is not going to be possible to start with unless
we provide some url unpacker method that takes a full url and turns
it back into baseurl, path and params

However there is still the issue of the tempaltes which require PageInfo to get the urls they need to spit out. One possible solution would be to give access to give the freemarker templates access to the buildURL function in some shape or form. Perhaps just adding a global function or just making it a function on page info? Something like:

${page.uri(page.pageURI)}

There PageInfo#uri would just be a call through to ResponseUtils.buildURL()

I can try to do that, just not sure how to simplify buildURL to
that point, see the above signature

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Justin_Deoliveira · September 8, 2009, 3:26pm

Andrea Aime wrote:

Justin Deoliveira ha scritto:

To sum up, what I'd need to properly handle the url
building with the URLManglers is:
- the meaning of the three URI above
- how to get the proper context path out of that restlet
request
One final thing I've noticed is that the URLs are built
piecemeal in various places in the rest code, that needs
to change as the url manglers can (and will in Tike case)
append kvp parameters to the URL.
So it's not possible to mangle the url in one place and
then adding path and kvp pieces of it later, everything
has to be available in the place where the final URL
is built.
It does not look too bad thought, the only place I
can see that builds urls is CatalogResourceBase.

Not sure I quite understand. Can the rest code just run the built url through the mangler extension point before it encodes it?

Yes and no. At the moment the code proxifies the URI
before building the PageInfo, and then appends stuff (formats)
to those proxified URIs in another class.
What needs doing is storing the original URI in the PageInfo
instead, and have the code building the final URL use
the originals, extract the expected base URL (or just have it,
like in the proposal above), add whatever else is needed to the path
portion and finally call ResponseUtil.buildURL

However that would make PageInfo something that is not a good
idea to pass over to the templates (as it would contain
un-proxied stuff). Suggestions?

So, just to ensure I understand the issue. The issue is that the rest dispatcher will first try to mangle the url, so in the case of proxying and the case (which I think is TIKE) append some kvp parameters. So taken the url:

http://host/geoserver/rest/foo.bar

It would mangle to:

http://proxyhost/geoserver/rest/foo.bar?key=value

Now... the problem is that when restconfig gets a hold of this to do resource liking it will turn it to:

http://proxyhost/geoserver/rest/foo/child.bar

Worse, it would end up just doing and appendPath which will result in:

http://proxyhost/geoserver/rest/foo.bar?key=value/child.bar

Cool, understood.

One possible solution is to have the restconfig just call buildURL again. I guess to have this work it might be a good idea to have the following always hold true:

buildURL(original) = buildURL(buildURL(original))

Mind buildURL takes quite a bit of parameters:

buildURL(String baseURL, String path, Map<String, String> kvp, URLType type)

As for having calls twice in a row be equivalent, that is not possible
as the kvp is embedded in the resulting path.
We'd need some way to re-parse is back separating the base url,
the path, and the kvp. As for having it work so that applying it
twice would lead to the same results we'd have to ask the
manglers to respect that rule.
But as I said, that is not going to be possible to start with unless
we provide some url unpacker method that takes a full url and turns
it back into baseurl, path and params

I wonder if it would be better to avoid the pre-mangling by the REST dispatcher. And instead make it mandatory for any restlet/resource that is outputing a url call PageInfo.uri(), which does the mangling. That should avoid the double case. And yes, it would seem to make sense to have a convenience method which takes a single string, and breaks it down into the components required for buildURL.

However there is still the issue of the tempaltes which require PageInfo to get the urls they need to spit out. One possible solution would be to give access to give the freemarker templates access to the buildURL function in some shape or form. Perhaps just adding a global function or just making it a function on page info? Something like:

${page.uri(page.pageURI)}

There PageInfo#uri would just be a call through to ResponseUtils.buildURL()

I can try to do that, just not sure how to simplify buildURL to
that point, see the above signature

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.