#160: CSW server sometimes gives ConcurrentModificationException
----------------------+-----------------------------------------------------
Reporter: simonp | Owner: geonetwork-devel@lists.sourceforge.net
Type: defect | Status: new
Priority: critical | Milestone: v2.4.1
Component: General | Version: v2.5.0
Keywords: |
----------------------+-----------------------------------------------------
Reported by James Q Wilson (james.q.wilson@...275...):
I've been trying to do some load tests on geonetwork 2_4_1 csw getRecords
service and I am getting java.util.ConcurrentModificationExceptions thrown
by Jeeves.
What I've done is loaded up 1000 test records onto the server, and written
a simple python client to sequentially fire CSW GetRecords queries at the
server, asking for a randomly generated bounding box. For a single client,
the server can handle ~ 60 requests / second.
If I fire up multiple instances of the client I occasionally get a
java.util.ConcurrentModificationException error thrown and the client
quits. I've spent some time looking through the code, and the error is
thrown somewhere within CatalogDispatcher.java.
Although I doubt a production server would be hit this hard, it does cause
some concern. Are there other subtle concurrency bugs out there waiting to
bite, that have an effect on data integrity, for example.
The obvious (and painful) fix is to make the app single threaded, by
synchronising the whole servlet (ie synchronize Jeeves doGet doPost). Of
course, this introduces significant latency when the server gets loaded.
Has anyone else seen errors of this type or does anybody have suggestions
as to how they could be tracked down - I don't have much expertise in
concurrency issues.
--
Ticket URL: <http://trac.osgeo.org/geonetwork/ticket/160>
GeoNetwork opensource Developer website <http://trac.osgeo.org/geonetwork>
GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. This website contains information related to the development of the software.
#160: CSW server sometimes gives ConcurrentModificationException
-----------------------+----------------------------------------------------
Reporter: simonp | Owner: geonetwork-devel@lists.sourceforge.net
Type: defect | Status: new
Priority: critical | Milestone: v2.4.1
Component: General | Version: v2.5.0
Resolution: | Keywords:
-----------------------+----------------------------------------------------
Comment (by simonp):
Or to put it more clearly: I've tested it and it doesn't happen when
hsListeners in DbmsPool.java is synchronized. But more analysis is
required.
Simon Pigot wrote:
> > Had a brief look at this in 2.4.1 using a couple of sessions that send
> > requests via curl.
> >
> > Saw the following exception:
> >
> > 2009-10-14 14:19:23,058 DEBUG [jeeves.service] - Raised exception
while
> > executing service
> > <error id="error">
> > <message />
> > <class>ConcurrentModificationException</class>
> > <stack>
> > <at class="java.util.HashMap$HashIterator" file="HashMap.java"
> > line="793" method="nextEntry" />
> > <at class="java.util.HashMap$KeyIterator" file="HashMap.java"
> > line="828" method="next" />
> > <at class="jeeves.resources.dbms.DbmsPool" file="DbmsPool.java"
> > line="173" method="close" />
> > <at class="jeeves.server.resources.ResourceManager"
> > file="ResourceManager.java" line="117" method="release" />
> > <at class="jeeves.server.resources.ResourceManager"
> > file="ResourceManager.java" line="83" method="close" />
> > <at class="jeeves.server.dispatchers.ServiceInfo"
> > file="ServiceInfo.java" line="245" method="execService" />
> > <at class="jeeves.server.dispatchers.ServiceInfo"
> > file="ServiceInfo.java" line="141" method="execServices" />
> > <at class="jeeves.server.dispatchers.ServiceManager"
> > file="ServiceManager.java" line="377" method="dispatch" />
> > <at class="jeeves.server.JeevesEngine" file="JeevesEngine.java"
> > line="621" method="dispatch" />
> > <at class="jeeves.server.sources.http.JeevesServlet"
> > file="JeevesServlet.java" line="174" method="execute" />
> > </stack>
> > <request>
> >
> > Is this what you are seeing James?
> >
> > Seems it can be fixed by making hsListeners into a Synchronized set in
> > DbmsPool.java - but this is a bit of shallow analysis because
curiously
> > it doesn't seem to happen in an earlier version of the BlueNetMEST
> > (jetty 5 vs jetty 6 maybe?) which we've backported the CSW 2.0.2
> > implementation into.
> >
> > Cheers,
> > Simon
--
Ticket URL: <http://trac.osgeo.org/geonetwork/ticket/160#comment:1>
GeoNetwork opensource Developer website <http://trac.osgeo.org/geonetwork>
GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. This website contains information related to the development of the software.
#160: CSW server sometimes gives ConcurrentModificationException
-----------------------+----------------------------------------------------
Reporter: simonp | Owner: geonetwork-devel@lists.sourceforge.net
Type: defect | Status: closed
Priority: critical | Milestone: v2.4.1
Component: General | Version: v2.5.0
Resolution: fixed | Keywords:
-----------------------+----------------------------------------------------
Changes (by simonp):
* status: new => closed
* resolution: => fixed
Comment:
Further analysis: src/org/fao/geonet/services/main/CswDispatcher.java was
opening Dbms resource explicitly to create new SettingManager rather than
using the one provided in the context via gc.getSettingManager.
Switch to using setting manager from context fixes the problem - no need
to synchronize hsListeners set in jeeves (although this behaviour is
interesting!).
Fixed in svn commit 5373
This did not appear in the backported CSW 2.0.2 implementation because the
service change that uses the settingmanager to check whether the CSW
service was on had not been backported!
--
Ticket URL: <http://trac.osgeo.org/geonetwork/ticket/160#comment:2>
GeoNetwork opensource Developer website <http://trac.osgeo.org/geonetwork>
GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. This website contains information related to the development of the software.