[Geoserver-users] Suddenly getting JDBC connection errors

RICK_WAYNE · January 7, 2014, 11:23pm

Hmm, apparently my GeoServer 2.2.5 install had too much to drink on New Year’s Eve. I noticed today that my main app was really flaky, it would run OK for awhile and then not serve any maps, or would respond to GetFeatureInfo but not to GetMap. I looked back in the logs and I saw some JDBC connection errors; as nearly as I can figure out, they started on Jan 1. Such as:

2014-01-07 16:16:56,339 ERROR [geoserver.ows] -
java.lang.RuntimeException: Unable to obtain connection: Cannot get a connection, pool error Timeout waiting for idle object
at org.geotools.jdbc.JDBCDataStore.createConnection(JDBCDataStore.java:1661)

[…]

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.geotools.data.jdbc.datasource.AbstractManageableDataSource.getConnection(AbstractManageableDataSource.java:46)
at org.geotools.jdbc.JDBCDataStore.createConnection(JDBCDataStore.java:1647)
… 97 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1134)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
… 100 more

A Google search led me to this page, which cites that error and gives as the fix “Set maxActive = -1 in your tomcat/jetty configuration file.” Whew! Good thing there’s only one of THOSE, I know right where to go! And anyway this error didn’t happen during deployment, it happened to a server that’s been running well for months. I tried restarting the various services under the “just one of those things” hypothesis, even rebooting the server. But it comes back pretty quickly.

But I don’t have any connection pooling set up in Tomcat, it’s all just straight out of GeoServer’s GUI (I’m running PostGIS 2.0).

Anyone have any ideas? This is really crippling my application. Haven’t recently updated Java or GeoServer, or mod_jk, or even Apache I don’t think.

Thanks much,

rw

Syrus_Mesdaghi · January 7, 2014, 11:39pm

Hello all, we have run into an issue that has been hard to track down so we’d appreciate an any suggestions!

Sometimes on a given server, GetFeatureInfo hit box shrinks down significantly to a few pixels which no longer covers the entire custom point symbology. This problem only occurs when we use do not use ‘sld:WellKnownName’ symbology. An example of a simpler SLD we use that we have noticed the problem is:

<?xml version="1.0" encoding="UTF-8"?>

<StyledLayerDescriptor version=“1.0.0”

xsi:schemaLocation=“http://www.opengis.net/sld http://schemas.opengis.net/sld/1.0.0/StyledLayerDescriptor.xsd”

xmlns=“http://www.opengis.net/sld” xmlns:ogc=“http://www.opengis.net/ogc”

xmlns:xlink=“http://www.w3.org/1999/xlink” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”>

canchas_de_futbol

image/png

25

Note that the problem is not always present. once it starts occurring, restarting tomcat fixes the problem. 90+ percent of the time the issue goes away after one restart but in some rare cases we have to restart it a few times.

When the problem starts happening, the wms layer stops rendering the symbols properly as well. When a feature happens to cross the boundaries of two tiles, the tile in which the very center of a particular feature falls will get the symbol rendered in it but the adjacent tile will not have the remaining portion of the symbol rendered (which can be up to about half of the symbol).

We have confirmed that the problem is not specific to our web client. If we go to layer preview and use OpenLayers to view the layer, even though the tiles render just fine (even if we select tiled), the hit box of the features are significantly smaller than they should be. Again, restarting tomcat fixes the issue so layer preview has the more forgiving hit box derived from the size of the largest symbol available in the sld of the layer. If we restart to fix the problem, in the near future the problem starts again… often in under 24 hours. Our web client is using ol3 but if we bring in the same layer from another server on which the problem has not started yet, one layer will be fine while other has problematic tiles and the features are harder to get info on. To fix the issue in ol2 layer preview, we can also start passing in parameter of buffer 15 through the URL, but this still doesn’t explain what is causing things to go wrong while a server is running. We have seen this problem on at least 6 different servers. The version of geoserver is:

Build Information

Version2.3-SNAPSHOT
Git Revision07e35af0864e92bf5c4e21681448d511c25a3c07
Build Date16-Sep-2013 23:36
GeoTools Version9-SNAPSHOT (rev 5c612984a088ef1e25de02c713359c4f904fef6b)

We have looked through geoserver logs for anything suspicious and haven’t found anything concerning. Another way to make the problem go away is to change the sld to use a wellknown symbol which is not an option for us.

We have also peeked through the following trying to narrow down the issue:

https://github.com/geoserver/geoserver/blob/2.3.x/src/wms/src/main/java/org/geoserver/wms/GetFeatureInfo.java#L408

looks like a decent amount of changes have been introduced in 2.4:

https://github.com/geoserver/geoserver/blob/2.4.x/src/wms/src/main/java/org/geoserver/wms/GetFeatureInfo.java#L412

Figured Justin or Andrea might know what is going on and perhaps this issue might have driven some of the changes in 2.4.

Thanks for any info!

Syrus Mesdaghi

Andrea_Aime4 · January 8, 2014, 7:31am

On Wed, Jan 8, 2014 at 12:39 AM, Syrus Mesdaghi <syrusm@anonymised.com> wrote:

We have also peeked through the following trying to narrow down the issue:

https://github.com/geoserver/geoserver/blob/2.3.x/src/wms/src/main/java/org/geoserver/wms/GetFeatureInfo.java#L408

looks like a decent amount of changes have been introduced in 2.4:

https://github.com/geoserver/geoserver/blob/2.4.x/src/wms/src/main/java/org/geoserver/wms/GetFeatureInfo.java#L412

Figured Justin or Andrea might know what is going on and perhaps this
issue might have driven some of the changes in 2.4.

No, this issue has not been reported before.
Worth mentioning that vector GetFeatureInfo has been completely rewritten
for the 2.5.x series (first beta should be released in 10 days)
to handle odd shaped symbols and, generally speaking, symbol sizes that
vary from feature to feature

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------