I am having a problem when two identical queries are issued to GeoServer at
the same time. Most of the time, these concurrent queries both return valid
results, but about 1 in 20 of them return erroneously due to a
java.util.ConcurrentModificationException.
I've seen mention of a similar problem in several GeoTools bugs, e.g.
GEOT-1271. However, this bug is marked as solved in GeoTools 2.2.3.
According to the GeoServer 1.4.1 README file, I have GeoTools
2.2.3-SNAPSHOT, so I think I should have this fix. However, I'm not sure if
there is a distinction between GeoTools 2.2.3 and 2.2.3-SNAPSHOT; it may
also be that this particular condition was not covered by the GEOT-1271 fix.
Any insights into this problem would be greatly appreciated. My system
configuration and log information are given below.
Thanks,
Steve
For reference, I am running the following:
GeoServer 1.4.1 in Jetty Servlet container
OS: Windows Server 2003
Web Server: IIS
DataStore: PostGIS database running on a separate (Linux) system
Log Messages from the error:
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: handling request:
Request: null
output format:GML2
max features:2147483647
version:
queries:
Query
feature type: wsi:latest_metar_reports
filter: [ site_id = KBOS ]
[properties: ALL ]
^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: handling request:
Request: null
output format:GML2
max features:2147483647
version:
queries:
Query
feature type: wsi:latest_metar_reports
filter: [ site_id = KBOS ]
[properties: ALL ]
^M
Jun 11, 2007 7:52:26 AM org.geotools.data.jdbc.QueryData finalize^M
SEVERE: There's code leaving readers or writers unclosed (you got an
unclosed QueryData object, which is usually held by a reader or a writer).
Call reader/writer.close() after using them to ensure they do not hold state
such as JDCB connections.
QueryData was open against feature type: latest_metar_reports^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: Service handled^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.ServiceException getXmlResponse^M
WARNING: encountered error: null error:Translator error
StackTrace: javax.xml.transform.TransformerException: Translator error^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:130)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:105)^M
at
org.vfny.geoserver.wfs.responses.GML2FeatureResponseDelegate.encode(GML2FeatureResponseDelegate.java:220)^M
at
org.vfny.geoserver.wfs.responses.FeatureResponse.writeTo(FeatureResponse.java:138)^M
at
org.vfny.geoserver.servlets.AbstractService.doService(AbstractService.java:598)^M
at
org.vfny.geoserver.servlets.AbstractService.doPost(AbstractService.java:457)^M
at org.geoserver.request.Dispatcher.post(Dispatcher.java:272)^M
at org.geoserver.request.Dispatcher.dispatch(Dispatcher.java:170)^M
at
org.geoserver.request.Dispatcher.handleRequestInternal(Dispatcher.java:56)^M
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)^M
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)^M
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)^M
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)^M
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)^M
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:357)^M
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)^M
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)^M
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)^M
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1050)^M
at
org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)^M
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1041)^M
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:354)^M
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)^M
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)^M
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)^M
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:123)^M
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)^M
at org.mortbay.jetty.Server.handle(Server.java:269)^M
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)^M
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:701)^M
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:617)^M
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)^M
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)^M
at
org.mortbay.jetty.nio.HttpChannelEndPoint.run(HttpChannelEndPoint.java:270)^M
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)^M
Caused by: java.util.ConcurrentModificationException^M
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)^M
at java.util.LinkedList$ListItr.next(LinkedList.java:696)^M
at
org.geotools.data.jdbc.ConnectionPool.getInUseManagedPooledConnection(ConnectionPool.java:150)^M
at
org.geotools.data.jdbc.ConnectionPool.access$200(ConnectionPool.java:46)^M
at
org.geotools.data.jdbc.ConnectionPool$ConnectionListManager.connectionClosed(ConnectionPool.java:227)^M
at
org.postgresql.ds.common.PooledConnectionImpl.fireConnectionClosed(PooledConnectionImpl.java:175)^M
at
org.postgresql.ds.common.PooledConnectionImpl$ConnectionHandler.invoke(PooledConnectionImpl.java:331)^M
at $Proxy0.close(Unknown Source)^M
at org.geotools.data.jdbc.JDBCUtils.close(JDBCUtils.java:136)^M
at org.geotools.data.jdbc.QueryData.close(QueryData.java:176)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:65)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:61)^M
at org.geotools.data.MaxFeatureReader.close(MaxFeatureReader.java:63)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.handleFeatureReader(FeatureTransformer.java:430)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.encode(FeatureTransformer.java:396)^M
at
org.geotools.xml.transform.TransformerBase$XMLReaderSupport.parse(TransformerBase.java:611)^M
at
org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:484)^M
at
org.geotools.xml.transform.TransformerBase$Task.run(TransformerBase.java:293)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:126)^M
... 34 more^M
---------^M
java.util.ConcurrentModificationException^M
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)^M
at java.util.LinkedList$ListItr.next(LinkedList.java:696)^M
at
org.geotools.data.jdbc.ConnectionPool.getInUseManagedPooledConnection(ConnectionPool.java:150)^M
at
org.geotools.data.jdbc.ConnectionPool.access$200(ConnectionPool.java:46)^M
at
org.geotools.data.jdbc.ConnectionPool$ConnectionListManager.connectionClosed(ConnectionPool.java:227)^M
at
org.postgresql.ds.common.PooledConnectionImpl.fireConnectionClosed(PooledConnectionImpl.java:175)^M
at
org.postgresql.ds.common.PooledConnectionImpl$ConnectionHandler.invoke(PooledConnectionImpl.java:331)^M
at $Proxy0.close(Unknown Source)^M
at org.geotools.data.jdbc.JDBCUtils.close(JDBCUtils.java:136)^M
at org.geotools.data.jdbc.QueryData.close(QueryData.java:176)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:65)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:61)^M
at org.geotools.data.MaxFeatureReader.close(MaxFeatureReader.java:63)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.handleFeatureReader(FeatureTransformer.java:430)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.encode(FeatureTransformer.java:396)^M
at
org.geotools.xml.transform.TransformerBase$XMLReaderSupport.parse(TransformerBase.java:611)^M
at
org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:484)^M
at
org.geotools.xml.transform.TransformerBase$Task.run(TransformerBase.java:293)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:126)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:105)^M
at
org.vfny.geoserver.wfs.responses.GML2FeatureResponseDelegate.encode(GML2FeatureResponseDelegate.java:220)^M
at org.vfny.geoserver.wfs.responses.FeatureRespQuery
feature type: wsi:domestic_notams
filter: [ site_id = KBOS ]
[properties: ALL ]
--
View this message in context: http://www.nabble.com/ConcurrentModificationException-in-GeoServer-1.4.1%2C-GeoTools-2.2.3-SNAPSHOT-tf3902127.html#a11062401
Sent from the GeoServer - Dev mailing list archive at Nabble.com.
Hi Steve,
I am not surprised to see this type of concurrency issue pop up. If it
is possible for you I would grab the latest 1.5.1 release and see if the
issue persists. GeoServer 1.4.x is mostly just in maintenance mode and
only crucial bug fixes are being back ported to it, so it will not have
a lot of the improvements and bug fixes that have gone into 1.5.x and
geotools 2.3.x.
Regardless of this... we have set up a concurrency / stress testing
suite for GeoServer wms. I believe if we set up a similar wfs request
suite backed by postgis we can probably replicate the problem. We will
be putting out 1.5.2 relatively soon, and we should probably schedule a
task to do this.
Andrea + others: any thoughts?
-Justin
Steve Marshall wrote:
I am having a problem when two identical queries are issued to GeoServer at
the same time. Most of the time, these concurrent queries both result valid
results, but about 1 in 20 of them return erroneously due to a
java.util.ConcurrentModificationException.
I've seen mention of this problem in several GeoTools bugs, e.g. GEOT-1271.
However, this bug is marked as solved in GeoTools 2.2.3. According to the
GeoServer 1.4.1 README file, I have GeoTools 2.2.3-SNAPSHOT, so I think I
should have this fix. However, I'm not sure if there is a distinction
between GeoTools 2.2.3 and 2.2.3-SNAPSHOT; it may also be that this
particular condition was not covered by the GEOT-1271 fix.
Any insights into this problem would be greatly appreciated. My system
configuration and log information are given below.
Thanks,
Steve
For reference, I am running the following:
GeoServer 1.4.1 in Jetty Servlet container
OS: Windows Server 2003
Web Server: IIS
Log Messages from the error:
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: handling request:
Request: null
output format:GML2
max features:2147483647
version:
queries:
Query
feature type: wsi:latest_metar_reports
filter: [ site_id = KBOS ]
[properties: ALL ]
^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: handling request:
Request: null
output format:GML2
max features:2147483647
version:
queries:
Query
feature type: wsi:latest_metar_reports
filter: [ site_id = KBOS ]
[properties: ALL ]
^M
Jun 11, 2007 7:52:26 AM org.geotools.data.jdbc.QueryData finalize^M
SEVERE: There's code leaving readers or writers unclosed (you got an
unclosed QueryData object, which is usually held by a reader or a writer).
Call reader/writer.close() after using them to ensure they do not hold state
such as JDCB connections.
QueryData was open against feature type: latest_metar_reports^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.servlets.AbstractService
doService^M
INFO: Service handled^M
Jun 11, 2007 7:52:26 AM org.vfny.geoserver.ServiceException getXmlResponse^M
WARNING: encountered error: null error:Translator error
StackTrace: javax.xml.transform.TransformerException: Translator error^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:130)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:105)^M
at
org.vfny.geoserver.wfs.responses.GML2FeatureResponseDelegate.encode(GML2FeatureResponseDelegate.java:220)^M
at
org.vfny.geoserver.wfs.responses.FeatureResponse.writeTo(FeatureResponse.java:138)^M
at
org.vfny.geoserver.servlets.AbstractService.doService(AbstractService.java:598)^M
at
org.vfny.geoserver.servlets.AbstractService.doPost(AbstractService.java:457)^M
at org.geoserver.request.Dispatcher.post(Dispatcher.java:272)^M
at org.geoserver.request.Dispatcher.dispatch(Dispatcher.java:170)^M
at
org.geoserver.request.Dispatcher.handleRequestInternal(Dispatcher.java:56)^M
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)^M
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)^M
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)^M
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)^M
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)^M
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:357)^M
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)^M
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)^M
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)^M
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1050)^M
at
org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)^M
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1041)^M
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:354)^M
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)^M
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)^M
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)^M
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:123)^M
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)^M
at org.mortbay.jetty.Server.handle(Server.java:269)^M
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)^M
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:701)^M
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:617)^M
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)^M
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)^M
at
org.mortbay.jetty.nio.HttpChannelEndPoint.run(HttpChannelEndPoint.java:270)^M
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)^M
Caused by: java.util.ConcurrentModificationException^M
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)^M
at java.util.LinkedList$ListItr.next(LinkedList.java:696)^M
at
org.geotools.data.jdbc.ConnectionPool.getInUseManagedPooledConnection(ConnectionPool.java:150)^M
at
org.geotools.data.jdbc.ConnectionPool.access$200(ConnectionPool.java:46)^M
at
org.geotools.data.jdbc.ConnectionPool$ConnectionListManager.connectionClosed(ConnectionPool.java:227)^M
at
org.postgresql.ds.common.PooledConnectionImpl.fireConnectionClosed(PooledConnectionImpl.java:175)^M
at
org.postgresql.ds.common.PooledConnectionImpl$ConnectionHandler.invoke(PooledConnectionImpl.java:331)^M
at $Proxy0.close(Unknown Source)^M
at org.geotools.data.jdbc.JDBCUtils.close(JDBCUtils.java:136)^M
at org.geotools.data.jdbc.QueryData.close(QueryData.java:176)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:65)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:61)^M
at org.geotools.data.MaxFeatureReader.close(MaxFeatureReader.java:63)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.handleFeatureReader(FeatureTransformer.java:430)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.encode(FeatureTransformer.java:396)^M
at
org.geotools.xml.transform.TransformerBase$XMLReaderSupport.parse(TransformerBase.java:611)^M
at
org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:484)^M
at
org.geotools.xml.transform.TransformerBase$Task.run(TransformerBase.java:293)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:126)^M
... 34 more^M
---------^M
java.util.ConcurrentModificationException^M
at
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761)^M
at java.util.LinkedList$ListItr.next(LinkedList.java:696)^M
at
org.geotools.data.jdbc.ConnectionPool.getInUseManagedPooledConnection(ConnectionPool.java:150)^M
at
org.geotools.data.jdbc.ConnectionPool.access$200(ConnectionPool.java:46)^M
at
org.geotools.data.jdbc.ConnectionPool$ConnectionListManager.connectionClosed(ConnectionPool.java:227)^M
at
org.postgresql.ds.common.PooledConnectionImpl.fireConnectionClosed(PooledConnectionImpl.java:175)^M
at
org.postgresql.ds.common.PooledConnectionImpl$ConnectionHandler.invoke(PooledConnectionImpl.java:331)^M
at $Proxy0.close(Unknown Source)^M
at org.geotools.data.jdbc.JDBCUtils.close(JDBCUtils.java:136)^M
at org.geotools.data.jdbc.QueryData.close(QueryData.java:176)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:65)^M
at
org.geotools.data.jdbc.JDBCFeatureReader.close(JDBCFeatureReader.java:61)^M
at org.geotools.data.MaxFeatureReader.close(MaxFeatureReader.java:63)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.handleFeatureReader(FeatureTransformer.java:430)^M
at
org.geotools.gml.producer.FeatureTransformer$FeatureTranslator.encode(FeatureTransformer.java:396)^M
at
org.geotools.xml.transform.TransformerBase$XMLReaderSupport.parse(TransformerBase.java:611)^M
at
org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:484)^M
at
org.geotools.xml.transform.TransformerBase$Task.run(TransformerBase.java:293)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:126)^M
at
org.geotools.xml.transform.TransformerBase.transform(TransformerBase.java:105)^M
at
org.vfny.geoserver.wfs.responses.GML2FeatureResponseDelegate.encode(GML2FeatureResponseDelegate.java:220)^M
at org.vfny.geoserver.wfs.responses.FeatureRespQuery
feature type: wsi:domestic_notams
filter: [ site_id = KBOS ]
[properties: ALL ]
--
Justin Deoliveira
The Open Planning Project
jdeolive@anonymised.com
Steve Marshall ha scritto:
I am having a problem when two identical queries are issued to GeoServer at
the same time. Most of the time, these concurrent queries both return valid
results, but about 1 in 20 of them return erroneously due to a
java.util.ConcurrentModificationException.
I've seen mention of a similar problem in several GeoTools bugs, e.g.
GEOT-1271. However, this bug is marked as solved in GeoTools 2.2.3. According to the GeoServer 1.4.1 README file, I have GeoTools
2.2.3-SNAPSHOT, so I think I should have this fix.
Nope. 2.2.3-SNAPHOST are as you commit releases that come before the
2.2.3 final. I fixed the bug you're citing between 1.5.0 and 1.5.1,
and that's the only release with that specific bug fix so far.
As Justin mentioned, it's there result of the initial stress testing we
did (thought to make that specific bug appear I had to hit Geoserver with 80 concurrent threads doing WMS requests, it never showed on my
PC with less than 50 concurrent threads).
Unless we have some hard request (aka, people paying for another release on the 1.4.x series), there won't be a 1.4.2. It's a matter of resource management, we cannot afford to work on three branches at the same time
(probably next week 1.6.0-beta1 will be released).
So, I'd suggest you try out 1.5.1 and let us know how it goes.
Btw, you did not mention with JDK you're using. Have you seen the
"Geosever in production" page in the user guide?
Cheers
Andrea