[Geoserver-users] GeoServer web-interface hangs (seemingly randomly)

GeoServer GeoServer User
1

Hi list,

We’ve been having trouble with GeoServer for a while, and the problem does not seem to go away.

There have been two (non-related? or are they related?) problems.

The first is where the web-interface stops working at times (the loading just hangs), and the second is where a WMS request hangs?

There could be a network or tomcat issue, but it seems to just happen with GeoServer, we’re running other services just fine.

We’re running GeoServer version 2.4.4 on tomcat 7.0.50. We’re using one PostGIS datastore and two Shapefile datastores (99.9% of the requests are to the PostGIS datastore).

The WMS request that hangs is usually (perhaps always) to a published feature type with point data (EPSG:4326) and has around 70k points stored. This only happens if the request has not been sent for a while (if it’s not stored in cache?). If the request is re-sendt instantly, it gets loaded instantly, or else we could wait and it might load after some minutes. Is it my configuration that could be wrong?
An example of request is: /wms?LAYERS=layerName&FORMAT=image%2Fpng&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG%3A4326&BBOX=-83.238653735919,6.761346264081,13.522692528162,103.52269252816&WIDTH=256&HEIGHT=256

Here is a short extraction of server-status in tomcat for when the web-interface stops working (or, it “works” but it hangs and loads after perhaps 15+ minutes): http://pastebin.com/CDTM6w5d
As you can see, the stage halts at S, which is service. This might start when I enter the “layers” list at web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage or the org.geoserver.web.data.resource.ResourceConfigurationPage - but it only happens sometimes and I can not confirm that the reason is entering one of these pages. This does not affect WMS/WFS queries, but all web-queries it seems.

Thanks for any help that could solve these problems. And I’d be happy to give more information if there is anything in particular that could help solve the issue.

Cheers,
Aleksander Vines

2

On Fri, Jun 13, 2014 at 3:28 PM, Aleksander Vines <aleksander.vines@anonymised.com

wrote:

Hi list,

We've been having trouble with GeoServer for a while, and the problem does
not seem to go away.

There have been two (non-related? or are they related?) problems.

The first is where the web-interface stops working at times (the loading
just hangs), and the second is where a WMS request hangs?

There could be a network or tomcat issue, but it seems to just happen with
GeoServer, we're running other services just fine.

We're running GeoServer version 2.4.4 on tomcat 7.0.50. We're using one
PostGIS datastore and two Shapefile datastores (99.9% of the requests are
to the PostGIS datastore).

The WMS request that hangs is usually (perhaps always) to a published
feature type with point data (EPSG:4326) and has around 70k points stored.
This only happens if the request has not been sent for a while (if it's not
stored in cache?). If the request is re-sendt instantly, it gets loaded
instantly, or else we could wait and it *might *load after some minutes.
Is it my configuration that could be wrong?
An example of request is: /wms?
LAYERS=layerName&FORMAT=image%2Fpng&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG%3A4326&BBOX=-83.238653735919,6.761346264081,13.522692528162,103.52269252816&WIDTH=256&HEIGHT=256

Here is a short extraction of server-status in tomcat for when the
web-interface stops working (or, it "works" but it hangs and loads after
perhaps 15+ minutes): http://pastebin.com/CDTM6w5d
As you can see, the stage halts at S, which is service. This might start
when I enter the "layers" list at
web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage or the
org.geoserver.web.data.resource.ResourceConfigurationPage - but it only
happens sometimes and I can not confirm that the reason is entering one of
these pages. This does not affect WMS/WFS queries, but all web-queries it
seems.

You are probably hitting the global configuration lock.
This lock is acquired any time a configuration page is generated, or an
action is called onto it, to avoid issues with concurrent modification of
the
geoserver configuratino (which is not thread safe, and not easy to make
thread safe).

So basically, every time you play with the REST api or the Web
configuration, a lock is acquired for the duration of the http call behind
it, and during
that time, no other request to the configuration subsystem can be made.
To give you an example, if you have multiple administrators, and one
decides to compute the bbox of a layer that has 100 million lines in a
database,
you GUI will likely be blocked for half an hour.

The way out of this would be to make the configuration code completely
thread safe, but it's not easy, and nobody funded this work so far.

The lock can be disabled via a java system variable,
-DGeoServerConfigurationLock.enabled=false,
but doing so will likely result in the configuration getting corrupted as
people edit it in parallel,
and a full "reload" is the only thing that will make you (at least
partially) recover it.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

3

Hi Andrea,

Thanks for the fast reply.

That does indeed sound like the reason of why the web-configuration is hanging.

However, it does not seem like the cause of the second problem(?), which is only occurring when there have not been any request to the WMS-server in a while. And this only apply to the WMS-requests, this is a bigger issue because this affects users of the system and not only administrators.

Though, it could seem like some sort of lock applies to these as well: atm the wms-requests have loaded for 1745875 ms. If I re send the request, it loads instantly, but the old request keeps “loading” and will eventually time out.

Many thanks,
Aleksander

Andrea Aime <andrea.aime@anonymised.com> , 6/13/2014 3:52 PM:

On Fri, Jun 13, 2014 at 3:28 PM, Aleksander Vines <aleksander.vines@anonymised.com> wrote:

Hi list,

We’ve been having trouble with GeoServer for a while, and the problem does not seem to go away.

There have been two (non-related? or are they related?) problems.

The first is where the web-interface stops working at times (the loading just hangs), and the second is where a WMS request hangs?

There could be a network or tomcat issue, but it seems to just happen with GeoServer, we’re running other services just fine.

We’re running GeoServer version 2.4.4 on tomcat 7.0.50. We’re using one PostGIS datastore and two Shapefile datastores (99.9% of the requests are to the PostGIS datastore).

The WMS request that hangs is usually (perhaps always) to a published feature type with point data (EPSG:4326) and has around 70k points stored. This only happens if the request has not been sent for a while (if it’s not stored in cache?). If the request is re-sendt instantly, it gets loaded instantly, or else we could wait and it might load after some minutes. Is it my configuration that could be wrong?
An example of request is: /wms?LAYERS=layerName&FORMAT=image%2Fpng&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&SRS=EPSG%3A4326&BBOX=-83.238653735919,6.761346264081,13.522692528162,103.52269252816&WIDTH=256&HEIGHT=256

Here is a short extraction of server-status in tomcat for when the web-interface stops working (or, it “works” but it hangs and loads after perhaps 15+ minutes): http://pastebin.com/CDTM6w5d
As you can see, the stage halts at S, which is service. This might start when I enter the “layers” list at web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage or the org.geoserver.web.data.resource.ResourceConfigurationPage - but it only happens sometimes and I can not confirm that the reason is entering one of these pages. This does not affect WMS/WFS queries, but all web-queries it seems.

You are probably hitting the global configuration lock.
This lock is acquired any time a configuration page is generated, or an action is called onto it, to avoid issues with concurrent modification of the
geoserver configuratino (which is not thread safe, and not easy to make thread safe).

So basically, every time you play with the REST api or the Web configuration, a lock is acquired for the duration of the http call behind it, and during
that time, no other request to the configuration subsystem can be made.
To give you an example, if you have multiple administrators, and one decides to compute the bbox of a layer that has 100 million lines in a database,
you GUI will likely be blocked for half an hour.

The way out of this would be to make the configuration code completely thread safe, but it’s not easy, and nobody funded this work so far.

The lock can be disabled via a java system variable, -DGeoServerConfigurationLock.enabled=false,
but doing so will likely result in the configuration getting corrupted as people edit it in parallel,
and a full “reload” is the only thing that will make you (at least partially) recover it.

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

4

On Mon, Jun 16, 2014 at 9:48 AM, Aleksander Vines <aleksander.vines@anonymised.com

wrote:

Though, it could seem like some sort of lock applies to these as well: atm
the wms-requests have loaded for 1745875 ms. If I re send the request, it
loads instantly, but the old request keeps "loading" and will eventually
time out.

This might be a connection pooling issue, recently discovered and being
discussed here:
http://osgeo-org.1560.x6.nabble.com/Upgrading-our-connection-pool-library-td5137354.html#a5143681

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------