[GeoNetwork-devel] [GeoNetwork opensource Developer website] #161: selection manager set iterators not synchronized, can fail with ConcurrentModificationException

#161: selection manager set iterators not synchronized, can fail with
ConcurrentModificationException
---------------------+------------------------------------------------------
Reporter: simonp | Owner: geonetwork-devel@lists.sourceforge.net
     Type: defect | Status: new
Priority: major | Milestone: v2.4.1
Component: General | Version: v2.5.0
Keywords: |
---------------------+------------------------------------------------------
Reported by James.Q.Wilson@...275...:

I'm discovering more concurrency errors under load in geonetworks. BTW -
thanks for CSW fix for error I reported earlier in week.

The errors are for geonetworks 2_4_1 in Tomcat 6.20 with 1000 iso records
loaded:

- Massive Delete fails with java.util.ConcurrentModificationException -
see stack trace below. This results in inconsistency between lucene index
and database. Lucene index drops records up to point at which exception
occurs. Since commit is never done in database, data still exists, and can
be recovered by rebuilding lucene index.
- MassiveUpdatePriviledges fails with
java.util.ConcurrentModificationException - see stack trace below
- MassiveUpdateCategories - got no stack trace recorded for this one.

Problem seems to be with HashSet declared in SelectionManager. These are
wrapped as synchronizedSet, which requires all iterators to be in
synchronized blocks (according to Java doc).

Iterators over this map in MassiveDelete / etc are not synchronized.
Explicitly synchronizing the iterators when these sets are accessed seems
to fix the concurrency problems:

eg

synchronized(sm.getSelection("metadata")) {

         // use iterator here
}

Regards

James

Stack traces:

First the massive delete:

88562 [34658350@...604...] ERROR jeeves.service - Exception when executing
servic
e
788562 [34658350@...604...] ERROR jeeves.service - (C) Exc :
java.util.Concurrent
ModificationException
788562 [34658350@...604...] DEBUG jeeves.service - Raised exception while
executin
g service
<error id="error">
   <message />
   <class>ConcurrentModificationException</class>
   <stack>
     <at class="java.util.HashMap$HashIterator" file="HashMap.java"
line="921" me
thod="nextEntry" />
     <at class="java.util.HashMap$KeyIterator" file="HashMap.java"
line="956" met
hod="next" />
     <at class="org.fao.geonet.services.metadata.MassiveDelete"
file="MassiveDele
te.java" line="87" method="exec" />
     <at class="jeeves.server.dispatchers.ServiceInfo"
file="ServiceInfo.java" li
ne="238" method="execService" />
     <at class="jeeves.server.dispatchers.ServiceInfo"
file="ServiceInfo.java" li
ne="141" method="execServices" />
     <at class="jeeves.server.dispatchers.ServiceManager"
file="ServiceManager.ja
va" line="377" method="dispatch" />
     <at class="jeeves.server.JeevesEngine" file="JeevesEngine.java"
line="621" m
ethod="dispatch" />
     <at class="jeeves.server.sources.http.JeevesServlet"
file="JeevesServlet.jav
a" line="174" method="execute" />
     <at class="jeeves.server.sources.http.JeevesServlet"
file="JeevesServlet.jav
a" line="89" method="doGet" />
     <at class="javax.servlet.http.HttpServlet" file="HttpServlet.java"
line="707
" method="service" />
   </stack>
   <request>
     <language>en</language>
     <service>metadata.massive.delete</service>
   </request>
</error>

Then the massive Privs:

1454281 [http-8080-3] ERROR jeeves.service - Exception when executing
service
1454281 [http-8080-3] ERROR jeeves.service - (C) Exc :
java.util.ConcurrentMod
ificationException
java.util.ConcurrentModificationException
         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:921)
         at java.util.HashMap$KeyIterator.next(HashMap.java:956)
         at
org.fao.geonet.services.metadata.MassiveUpdatePrivileges.exec(Massive
UpdatePrivileges.java:85)
         at
jeeves.server.dispatchers.ServiceInfo.execService(ServiceInfo.java:24
4)
         at
jeeves.server.dispatchers.ServiceInfo.execServices(ServiceInfo.java:1
41)
         at
jeeves.server.dispatchers.ServiceManager.dispatch(ServiceManager.java
:377)
         at jeeves.server.JeevesEngine.dispatch(JeevesEngine.java:621)
         at
jeeves.server.sources.http.JeevesServlet.execute(JeevesServlet.java:1
74)
         at
jeeves.server.sources.http.JeevesServlet.doGet(JeevesServlet.java:89)

         at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
         at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
         at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
         at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)
         at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)
         at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:128)
         at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)
         at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)
         at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:293)
         at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:849)
         at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:583)
         at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:45
4)
         at java.lang.Thread.run(Thread.java:619)

--
Ticket URL: <http://trac.osgeo.org/geonetwork/ticket/161&gt;
GeoNetwork opensource Developer website <http://trac.osgeo.org/geonetwork&gt;
GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. This website contains information related to the development of the software.

#161: selection manager set iterators not synchronized, can fail with
ConcurrentModificationException
----------------------+-----------------------------------------------------
  Reporter: simonp | Owner: geonetwork-devel@lists.sourceforge.net
      Type: defect | Status: closed
  Priority: major | Milestone: v2.4.1
Component: General | Version: v2.5.0
Resolution: fixed | Keywords:
----------------------+-----------------------------------------------------
Changes (by simonp):

  * status: new => closed
  * resolution: => fixed

Comment:

Also apply to SelectionSearch, MassiveNewOwner and MassiveXslProcessing

Fixed in svn commit rev 5374

--
Ticket URL: <http://trac.osgeo.org/geonetwork/ticket/161#comment:1&gt;
GeoNetwork opensource Developer website <http://trac.osgeo.org/geonetwork&gt;
GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web. It provides powerful metadata editing and search functions as well as an embedded interactive web map viewer. This website contains information related to the development of the software.