CSP with a proxy, how to deal with it

GeoServer GeoServer Developer
1

Hi all,
I’ve spent some time trying to figure out how to attack the issue of CSP + Proxy with SSL termination, this mail contains my findings (warning, long and technical).

First of all, what’s going on?
GeoServer code, Spring security, and Wicket internals, all tend to use HTTPServletReponse.redirect(“${relativePath}”), where relativePath is often “web”.

From Servlet API javadoc (thanks Steve) we learn the method should be implemented by the container by turning the relative path into an absolute URL.
That is done by taking the HTTP Host header, which is part of the spec and guaranteed to be there, the context path (e.g., “geoserver”) and the relative path provided as an argument.

The protocol is an issue: there is not true standard HTTP header indicating the protocol (there are non-standard headers though, more on this later), so the servlet container will use whatever protocol it received the request onto.

Now, it’s pretty common practice that SSL is handled by proxy in front of GeoServer, with nginx and apache being common choices. So the request goes to, say “https://gs-main.geosolutionsgroup.com/geoserver/web/”, the proxy handles SSL and forwards the request as “http://:8080/geoserver/web/” and when doing a redirect the servlet container returns a HTTP 302 with a Location header stating “http://gs-main.geosolutionsgroup.com/geoserver/myChosenPath”. Whoops!

So far everything worked anyways, because most sites have transitioned in recent years from HTTP to HTTPS and have a built-in configuration redirecting any HTTP request to HTTPS.
So it works by doing this:

  1. GeoServer sends a redirect with Location header set to http://gs-main.geosolutionsgroup.com/geoserver/myChosenPath
  2. The browser receives exactly that, and opens that new URL
  3. The proxy in turns performs another redirect, this time to https://gs-main.geosolutionsgroup.com/geoserver/myChosenPath
  4. The browser follows the new path and we’re back in business
    This happens on a set of very common actions in the GeoServer UI: login, logout, switch tab in the tabbed layer pages, click save in a number of pages where save is implemented as an Ajax button.

Introduce CSP, and the browser gets instructed not to reach out to pages outside of the origin server and protocol, and our little set of actions above stops at point 2., with errors only visible if you open the developer tools.

How to get out of it? I found and considered a few approaches.

2

Sigh, bloody Discord cut my message… here is the rest of it:

First one: have GeoServer redirect to a full URL every time. That’s allowed by the servlet API and we could build the full redirect URL by using the proxy base URL configuration. Redirects are done also in code outside of GeoServer, but this could be performed in a servlet wrapper.

However, I know for a fact it’s going to break secure deploys that don’t expose the GUI on the web… the typical situation there is that they access GeoServer directly within the DMZ, on the actual GeoServer IP, and the GUI needs to keep on working in that case.

We’d need some way to tell apart the local direct access from the “through proxy” access.

Which would mean, fiddling with the proxy configuration anyways. Meh!

Second one: tell the web container which protocol to use. The non standard, yet common, “X-Forwarded-Proto” header can be set to “https” as a way to tell the container which protocol it should use for redirects. However, containers will, out of the box, ignore it, because the header is not guarnteed to be there and it may be manipulated by clients to mount some sort of attack.

Futhermore, each container has a different way to configure using the header. For example, in Tomcat it would be declaring a Valve in server.xml:

But in Jetty, it would be something completely different in start.ini:

–module=forwarded

(mind, untested) and so on for each different servlet container.

Third one: tell the proxy to switch the protocol

The proxy can be configured to switch the proxy in redirect calls. Again, we’re facing a case by case scenario.

For nginx (the case that I’ve actually tried, and can confirm it works):

proxy_redirect http://$host/ https://$host/;

For apache (mind, untested):

Header edit Location ^http://(.*)$ https://$1

and so on, a different way to do this in each proxy software.

Which one to choose? I’ve made a few questions around and asked people about this, as well as a few AIs, the summary is:

  • SSL termination is normally in the hands of system admins, which may or may not dealing with the GeoServer setup
  • Those same people are familiar with their proxy of choice, but very often, only lightly familiar with Tomcat configuration. Asking around about the Valve, I could not find anyone that knew what it was, while the nginx setting was known, although not normally used
  • Asking AIs confirmed that web container setup is well known only in Java heavy shops, but a number of GeoServer deploys are in environments where GS is the only Java app around.

So I’d suggest to go an document how to configure proxies. I can offer an example of Nginx.

Can anyone else test the Apache approach? And then we can ask users to contribute approaches for other proxies, once the admins know what to do, they should be able to figure out the specific setting quickly.

Regards,

Andrea Aime

3

Hi Andrea,

I will try to look at this in more detail later, but I’m under the impression we could
save us from a lot of pain if we do as spring-boot does when configured with
server.forward-headers-strategy=FRAMEWORK [1]

That is, setting up a ForwardedHeaderFilter, will take care of grabbing the X-Forwarded-Host/Port/Proto/Prefix/Ssl/For

request headers, and proceed with an HttpServletRequest wrapper that makes the rest of the filter chain reflect the
client-originated protocol and address.

This is what we do in GeoServer Cloud, every service application configuration sets that config property. So that for example,
the application context is always /, but the gateway (the reverse proxy) sends the appropriate X-Forwarded-* headers, and
GeoServer doesn’t need to do any URL mangling.

Hope that helps, I might be off topic, just replying in a rush in case it’s useful

[1] https://docs.spring.io/spring-boot/docs/2.7.15/reference/html/howto.html#howto.webserver.use-behind-a-proxy-server

4

@aaime-geosolutions thanks for continuing to peruse this topic.

Documenting how to configure proxies would make an excellent addition to running in a production environment.

I also had a look CSP topic week, to double check that the user docs provided some guidance on use of org.geoserver.web.csp.strict=false to aid with troubleshooting … and found we did not have have an entry on “user interface” troubleshooting.

The resulting PR has updated:

  • Troubleshooting mentioning both “Oops, something went wrong” stack traces, and “User interface not responsive” for CSP issues.
  • I also rounded up all the CSP application property settings which were previously hidden in different tutorials.
5

Hi Gabriel,
I like the idea, but we have to make sure that the proto header is not processed unless we’re in a
proxied environment, a directly exposed GeoServer should ignore it… so it would have to be something
programmatically controllable, not always on (a subclass of that filter, implementing GeoServerFilter?)

The most immediate approach would be to check if proxy base URL is set or not… if in a proxied environment,
then start caring about the proto header.

And if a request comes directly from the DMZ, then it should not have the header, done.
Can anyone thing of a configuration where the proxy base url is set, and yet someone can do direct requests,
manipulate the proto header to cause some damage? I guess that would imply already having a foothold in the DMZ…

Thoughts?