[GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

GeoNetwork GeoNetwork Developer
1

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system crashes.
What is really surprising is that many of those openfiles are tagged (deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes. We tried the "optimize lucene index" in the Admin interfaces but nothing changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271 1902329 /usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749 16352071 /usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391 16352010 /usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs (deleted)
java 23332 root 107r REG 253,0 10081 1902326 /usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs (deleted)
java 23332 root 108r REG 253,0 7471 16352059 /usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs (deleted)
...

# Could there be a relation with the other issue I raised to the list today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

2

Sylvain,
maybe point #6
http://geonetwork-opensource.org/manuals/trunk/eng/users/admin/advanced-configuration/index.html#advanced-configuration-for-larger-catalogs
could help.

Ciao.

Francois

2012/6/20 Sylvain Grellet <s.grellet@anonymised.com>:

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

3

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

4

François, Simon,

Thanks for your answers. I had already raised the max opened files but not up to 4096 as mentioned in the link provided by François. I'll give it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

5

François, Simon,

A feedback on this aspect after a couple month with a max opened files raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork 3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on Geonetwork_opened_files_third_peak_detail.png), so it seems there is some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex : like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

(attachments)

Geonetwork_opened_files_third_peak_detail.png
Geonetwork_opened_files.png

6

Hi type of requests are most typically used? What version of geonetwork?

On Thu, Oct 4, 2012 at 5:52 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

François, Simon,

A feedback on this aspect after a couple month with a max opened files raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork 3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on Geonetwork_opened_files_third_peak_detail.png), so it seems there is some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex : like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@anonymised.comsourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

7

Hi Jesse,

Geonetwork : 2.6.4

As for the type of requests : a lot of different csw (but no CSWT). No OAI-PMH, no Z39.50.
All “CSW Discovery Interface” methods are being used.
Our CSW endpoint is used by many other organizations so I’m not logging in details all that goes in.

We also consume a lot our own CSW for various websites we host here.
The only unexpected behavior I have noticed with our CSW requests is described in another e-mail to the devel list : http://osgeo-org.1560.n6.nabble.com/CSW-XPath-and-Logging-issue-tt4982770.html
But I’m not really sure there is a connexion between both issues.

Thanks

Sylvain

···

On 05/10/2012 06:16, Jesse Eichar wrote:

Hi type of requests are most typically used? What version of geonetwork?

On Thu, Oct 4, 2012 at 5:52 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

François, Simon,

A feedback on this aspect after a couple month with a max opened files raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork 3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on Geonetwork_opened_files_third_peak_detail.png), so it seems there is some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex : like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

8

Yes CSW was one of the sources of problems. I have added a fix on all branches that fix several of the problems I have experienced with CSW. Like you we use CSW very heavily as well. It could be that my fixes will solve the issue, although I can’t guarantee it since I have only tried in my environment which is based on 2.8.x rather than the 2.6 line.

the two commits that I refer to are:

https://github.com/geonetwork/core-geonetwork/commit/7eff50acd7b24311e65f7e3734f8993897f24bb7
and
https://github.com/geonetwork/core-geonetwork/commit/369cef47b526e79ff3f0497900cf133960e04ea6

Jesse

On Fri, Oct 5, 2012 at 2:15 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

Hi Jesse,

Geonetwork : 2.6.4

As for the type of requests : a lot of different csw (but no CSWT). No OAI-PMH, no Z39.50.
All “CSW Discovery Interface” methods are being used.
Our CSW endpoint is used by many other organizations so I’m not logging in details all that goes in.

We also consume a lot our own CSW for various websites we host here.
The only unexpected behavior I have noticed with our CSW requests is described in another e-mail to the devel list : http://osgeo-org.1560.n6.nabble.com/CSW-XPath-and-Logging-issue-tt4982770.html
But I’m not really sure there is a connexion between both issues.

Thanks

Sylvain

On 05/10/2012 06:16, Jesse Eichar wrote:

Hi type of requests are most typically used? What version of geonetwork?

On Thu, Oct 4, 2012 at 5:52 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

François, Simon,

A feedback on this aspect after a couple month with a max opened files raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork 3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on Geonetwork_opened_files_third_peak_detail.png), so it seems there is some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex : like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

9

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

10

Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I'm pretty certain the problem is that which we ran across - I'll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we'll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com; fx.prunayre@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

11

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com> wrote:

Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com; fx.prunayre@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.com9…]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com
Cc: geonetwork-devel@lists.sourceforge.net; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.com]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

12

Hi Jesse,

Thanks. I've made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I'm pretty certain the problem is that which we ran across - I'll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we'll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com.31…>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.com.sourceforge.net>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comrceforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

13

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn’t able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com> wrote:

Hi Jesse,

Thanks. I’ve made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Saturday, 6 October 2012 4:13 AM

To: Pigot, Simon (CMAR, Hobart)

Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com192…mailto:[Simon.Pigot@anonymised.com2...](mailto:Simon.Pigot@anonymised.com)> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM

To: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)

Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.commailto:[s.grellet@anonymised.com.39...](mailto:s.grellet@anonymised.com)]

Sent: Friday, 5 October 2012 1:52 AM

To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com..)
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net); Sylvain GRELLET

Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com.)]

Sent: Thursday, 21 June 2012 1:13 AM

To: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.comt](mailto:geonetwork-devel@lists.sourceforge.net)

Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

14

Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Monday, 8 October 2012 6:24 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn't able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…>> wrote:
Hi Jesse,

Thanks. I've made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…<mailto:Simon.Pigot@anonymised.com>>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I'm pretty certain the problem is that which we ran across - I'll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we'll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com…><mailto:fx.prunayre@anonymised.com>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com.31…><mailto:fx.prunayre@anonymised.com>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.com.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comrceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

15

Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.com> wrote:

Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,

Simon

From: Jesse Eichar [jesse.eichar@…189…]

Sent: Monday, 8 October 2012 6:24 PM

To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.come.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn’t able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com192…mailto:[Simon.Pigot@anonymised.com2...](mailto:Simon.Pigot@anonymised.com)> wrote:
Hi Jesse,

Thanks. I’ve made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)]

Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)

Cc: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com); geonetwork-devel@anonymised.comourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comeforge.net)

Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com192…<mailto:Simon.Pigot@anonymised.com2…mailto:[Simon.Pigot@anonymised.com192...](mailto:Simon.Pigot@anonymised.com)>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM

To: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>

Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.commailto:[s.grellet@anonymised.com.39...](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.com.39…mailto:[s.grellet@anonymised.com..](mailto:s.grellet@anonymised.com)>]

Sent: Friday, 5 October 2012 1:52 AM

To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com..)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com.31...)>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>; Sylvain GRELLET

Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com.)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>]

Sent: Thursday, 21 June 2012 1:13 AM

To: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.comt](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.comet](mailto:geonetwork-devel@lists.sourceforge.net)>

Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.com.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

16

Seeing that there are no more deleted open resources in the list of resources, I would like to port this change to master. Is that ok?

On geocat.ch production server I did:

$ lsof -p 10582 | grep delete | wc -l
1982

that is a lot of wasted resources.

Jesse

On Mon, Oct 8, 2012 at 9:58 AM, Jesse Eichar <jesse.eichar@anonymised.com> wrote:

Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.com> wrote:

Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,

Simon

From: Jesse Eichar [jesse.eichar@anonymised.com]

Sent: Monday, 8 October 2012 6:24 PM

To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com…; fx.prunayre@anonymised.com; geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn’t able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.commailto:[Simon.Pigot@...192...](mailto:Simon.Pigot@anonymised.com)> wrote:
Hi Jesse,

Thanks. I’ve made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)]

Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)

Cc: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com); geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.comrge.net](mailto:geonetwork-devel@lists.sourceforge.net)

Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@…192…mailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM

To: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@anonymised.come.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comt)>

Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com.39...)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com.](mailto:s.grellet@anonymised.com)>]

Sent: Friday, 5 October 2012 1:52 AM

To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.com…31…mailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@anonymised.come.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comt)>; Sylvain GRELLET

Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>]

Sent: Thursday, 21 June 2012 1:13 AM

To: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@anonymised.comists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.com.sourceforge.net)>

Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comt](mailto:GeoNetwork-devel@anonymised.comts.sourceforge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@anonymised.comrceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comorge.net)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@anonymised.comrge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comnet)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list

GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@anonymised.comrge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comnet)>

https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

17

Yep go ahead and port it - thanks.

Cheers,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Monday, 8 October 2012 7:10 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

Seeing that there are no more deleted open resources in the list of resources, I would like to port this change to master. Is that ok?

On geocat.ch<http://geocat.ch> production server I did:

$ lsof -p 10582 | grep delete | wc -l
1982

that is a lot of wasted resources.

Jesse

On Mon, Oct 8, 2012 at 9:58 AM, Jesse Eichar <jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>> wrote:
Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…>> wrote:
Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>]
Sent: Monday, 8 October 2012 6:24 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn't able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…<mailto:Simon.Pigot@anonymised.com>>> wrote:
Hi Jesse,

Thanks. I've made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com><mailto:jesse.eichar@anonymised.com>]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com…><mailto:fx.prunayre@anonymised.com>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.com.sourceforge.net>>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com…192…<mailto:Simon.Pigot@anonymised.com><mailto:Simon.Pigot@anonymised.com>>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I'm pretty certain the problem is that which we ran across - I'll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we'll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com39...><mailto:s.grellet@anonymised.com>>; fx.prunayre@...31...<mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com.><mailto:fx.prunayre@anonymised.com>>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.com.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com<mailto:s.grellet@anonymised.com..>>>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com.31…><mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comurceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.com.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com.<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com39...>>>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.com.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.comrge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comrceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comnet<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comet<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comists.sourceforge.net<mailto:GeoNetwork-devel@lists.sourceforge.net>>>
geonetwork-devel List Signup and Options
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comts.sourceforge.net<mailto:GeoNetwork-devel@lists.sourceforge.net>>>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!

_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comts.sourceforge.net<mailto:GeoNetwork-devel@lists.sourceforge.net>>>

GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

18

Hi guys,

Just summarizing this thread to be sure I got it right & test it :

- patch proposed by Jesse has been committed either to 2.6.x branch & 2.8
=> no need for me to include github sources pointed by Jesse on his October 5th e-mail.

- Simon did you commit to the svn as well or should I get the updated 2.6.x branch from github only ?

Cheers.
Sylvain

On 08/10/2012 16:14, Simon.Pigot@anonymised.com wrote:

Yep go ahead and port it - thanks.

Cheers,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Monday, 8 October 2012 7:10 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comceforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

Seeing that there are no more deleted open resources in the list of resources, I would like to port this change to master. Is that ok?

On geocat.ch<http://geocat.ch> production server I did:

$ lsof -p 10582 | grep delete | wc -l
1982

that is a lot of wasted resources.

Jesse

On Mon, Oct 8, 2012 at 9:58 AM, Jesse Eichar <jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>> wrote:
Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com>> wrote:
Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com>]
Sent: Monday, 8 October 2012 6:24 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn't able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com>> wrote:
Hi Jesse,

Thanks. I've made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon
________________________________________
From: Jesse Eichar [jesse.eichar@anonymised.com<mailto:jesse.eichar@anonymised.com><mailto:jesse.eichar@anonymised.com>]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com..<mailto:s.grellet@anonymised.com>>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com31…><mailto:fx.prunayre@anonymised.com>; geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comts.sourceforge.net>>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com>>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I'm pretty certain the problem is that which we ran across - I'll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we'll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon
________________________________________
From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com..<mailto:s.grellet@anonymised.com>><mailto:s.grellet@anonymised.com..39...><mailto:s.grellet@anonymised.com>>; fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com...><mailto:fx.prunayre@anonymised.com>>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comsourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.comge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

- GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
- IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
- When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
- Lucene is a file based index, rewriting of index deletes some files and creates new ones
- it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as 'deleted'.

Cheers,
Simon

________________________________________
From: Sylvain GRELLET [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com<mailto:s.grellet@anonymised.com9...>>>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.com<mailto:fx.prunayre@…31…><mailto:fx.prunayre@anonymised.com><mailto:fx.prunayre@anonymised.com<mailto:fx.prunayre@anonymised.com..<mailto:fx.prunayre@anonymised.com>>>
Cc: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.comsourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.comge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I'll give
it a try.

In case this does not work, I'll try changing the Lucene version.

I'll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene *and* the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn't been reported there as a bug and it's interesting that you hadn't run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon
________________________________________
From: Sylvain Grellet [s.grellet@anonymised.com<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com..<mailto:s.grellet@anonymised.com><mailto:s.grellet@anonymised.com.39...>>>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.net<mailto:geonetwork-devel@anonymised.coms.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@anonymised.comorge.net<mailto:geonetwork-devel@lists.sourceforge.net><mailto:geonetwork-devel@lists.sourceforge.net>>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java "1.7.0_04".

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this "threshold", it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof....

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the "optimize lucene index" in the Admin interfaces but nothing
changed.
We updated java, nothing..

# lsof | grep geonetwork output
...
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
...

# Could there be a relation with the other issue I raised to the list
today ("CSW / XPath and Logging issue") ?

Thanks
Sylvain

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comurceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.com.net<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comrceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comnet<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comet<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comists.sourceforge.net<mailto:GeoNetwork-devel@lists.sourceforge.net>>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net<mailto:GeoNetwork-devel@anonymised.comceforge.net><mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comet<mailto:GeoNetwork-devel@lists.sourceforge.net><mailto:GeoNetwork-devel@anonymised.comists.sourceforge.net<mailto:GeoNetwork-devel@lists.sourceforge.net>>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

19

2.6.x and 2.8.x are quite different with regards to indexing so it will take me a little while to port the changes. But I am starting on this project. Although I must say that it will take me a little while still because I don’t have time right now at work to do this. I will let you know as soon as I have a patch for master and ready for testing.

Jesse

On Mon, Oct 8, 2012 at 5:15 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

Hi guys,

Just summarizing this thread to be sure I got it right & test it :

  • patch proposed by Jesse has been committed either to 2.6.x branch & 2.8
    => no need for me to include github sources pointed by Jesse on his October 5th e-mail.

  • Simon did you commit to the svn as well or should I get the updated 2.6.x branch from github only ?

Cheers.
Sylvain

On 08/10/2012 16:14, Simon.Pigot@anonymised.com wrote:

Yep go ahead and port it - thanks.

Cheers,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Monday, 8 October 2012 7:10 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com…; fx.prunayre@anonymised.com; geonetwork-devel@anonymised.comsourceforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

Seeing that there are no more deleted open resources in the list of resources, I would like to port this change to master. Is that ok?

On geocat.ch<http://geocat.ch> production server I did:

$ lsof -p 10582 | grep delete | wc -l
1982

that is a lot of wasted resources.

Jesse

On Mon, Oct 8, 2012 at 9:58 AM, Jesse Eichar <jesse.eichar@anonymised.commailto:[jesse.eichar@camptocamp.com](mailto:jesse.eichar@anonymised.com)> wrote:
Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com92...](mailto:Simon.Pigot@anonymised.com)> wrote:
Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@camptocamp.com](mailto:jesse.eichar@anonymised.com)]
Sent: Monday, 8 October 2012 6:24 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com…mailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com); geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn’t able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com92…mailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)>> wrote:
Hi Jesse,

Thanks. I’ve made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@camptocamp.com](mailto:jesse.eichar@anonymised.com)<mailto:jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com189...)>]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com…mailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>; geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@anonymised.comet)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.comsourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.com92…mailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)<mailto:Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)>>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.com…mailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)><mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com...](mailto:fx.prunayre@anonymised.com)>>
Cc: geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.com.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@anonymised.com39…netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comceforge.net)<mailto:geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)>>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)><mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com...](mailto:fx.prunayre@anonymised.com)>>
Cc: geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.com.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@anonymised.com39…netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comceforge.net)<mailto:geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)>>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@anonymised.com.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@anonymised.com39…netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@anonymised.comceforge.net)<mailto:geonetwork-devel@anonymised.comsourceforge.netmailto:[geonetwork-devel@anonymised.comnet](mailto:geonetwork-devel@lists.sourceforge.net)>>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comsourceforge.net](mailto:GeoNetwork-devel@anonymised.comsts.sourceforge.net)><mailto:GeoNetwork-devel@anonymised.com.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comrge.net)<mailto:GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comsourceforge.net](mailto:GeoNetwork-devel@anonymised.comsts.sourceforge.net)><mailto:GeoNetwork-devel@anonymised.com.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comrge.net)<mailto:GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comsourceforge.net](mailto:GeoNetwork-devel@anonymised.comsts.sourceforge.net)><mailto:GeoNetwork-devel@anonymised.com.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comrge.net)<mailto:GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@anonymised.comsourceforge.net](mailto:GeoNetwork-devel@anonymised.comsts.sourceforge.net)><mailto:GeoNetwork-devel@anonymised.com.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@anonymised.comrge.net)<mailto:GeoNetwork-devel@anonymised.comsourceforge.netmailto:[GeoNetwork-devel@anonymised.comnet](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

20

Ok,
So changes committed by Simon on October 7th (from BlueNetMEST 1.4.9 to 2.6.4) need to be enhanced by Jesse’s patch before testing.
Is this understanding correct ?

Sylvain

···

On 08/10/2012 17:41, Jesse Eichar wrote:

2.6.x and 2.8.x are quite different with regards to indexing so it will take me a little while to port the changes. But I am starting on this project. Although I must say that it will take me a little while still because I don’t have time right now at work to do this. I will let you know as soon as I have a patch for master and ready for testing.

Jesse

On Mon, Oct 8, 2012 at 5:15 PM, Sylvain GRELLET <s.grellet@anonymised.com> wrote:

Hi guys,

Just summarizing this thread to be sure I got it right & test it :

  • patch proposed by Jesse has been committed either to 2.6.x branch & 2.8
    => no need for me to include github sources pointed by Jesse on his October 5th e-mail.

  • Simon did you commit to the svn as well or should I get the updated 2.6.x branch from github only ?

Cheers.
Sylvain

On 08/10/2012 16:14, Simon.Pigot@anonymised.com wrote:

Yep go ahead and port it - thanks.

Cheers,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.com]
Sent: Monday, 8 October 2012 7:10 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.com; fx.prunayre@anonymised.com; geonetwork-devel@lists.sourceforge.net
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

Seeing that there are no more deleted open resources in the list of resources, I would like to port this change to master. Is that ok?

On geocat.ch<http://geocat.ch> production server I did:

$ lsof -p 10582 | grep delete | wc -l
1982

that is a lot of wasted resources.

Jesse

On Mon, Oct 8, 2012 at 9:58 AM, Jesse Eichar <jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)> wrote:
Here is my analysis of the open files:

lsof -p 35270 | grep lucene/nonspatial > /tmp/resources
grep -ie deleted /tmp/resources

The grep did not find any lines. So I am guessing the deleted issue is fixed. I did run optimize and that did not reduce the number of open resources.

On the one hand it looks good because the deleted issue seems resolved. However I am still concerned about the growing index. I admit that it might not be representative of normal usage but I am curious if it stops growing ever.

Jesse

On Mon, Oct 8, 2012 at 9:38 AM, <Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)> wrote:
Depends on what the open files are too - are they (deleted) files from the index? May be worthwhile running the Index Optimizer in Admin menu after the test to if they are to see whether any index changes go away (this runs once a day by default from memory).

Cheers,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)]
Sent: Monday, 8 October 2012 6:24 PM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com); geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I just simulated 20 users. Made > 200 requests (csw search) and the number of resources stayed steady.

My Jmeter tests use Csw-transaction for testing editing so I wasn’t able to test that. But I did use selenium to script adding and then deleting metadata. I have added at least 130 metadata and deleted them as well. Unfortunately the open file count has gone up from 750 to 1135. Not definitive proof that anything is wrong but it is worrying to me still. I will try to update jmeter to use the mef.import.ui service for adding metadata and see if I can run this test for a couple hours to see if the trend continues or if it eventually levels out.

Jesse

On Sun, Oct 7, 2012 at 4:53 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)>> wrote:
Hi Jesse,

Thanks. I’ve made the initial commit with the changes to the 2.6.x branch. Looks like we could use your jmeter module on the CSW server. I have a few other things to check with this as well.

Cheers and thanks,
Simon

From: Jesse Eichar [jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)<mailto:jesse.eichar@anonymised.commailto:[jesse.eichar@anonymised.com](mailto:jesse.eichar@anonymised.com)>]
Sent: Saturday, 6 October 2012 4:13 AM
To: Pigot, Simon (CMAR, Hobart)
Cc: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>; geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>
Subject: Re: [GeoNetwork-devel] [ExternalEmail] Re: GN 2.6.4 / Java 1.7 / Too many open files error

I will happily do what I can to help. Just let me know what I can do.

Jesse

On Fri, Oct 5, 2012 at 5:48 PM, <Simon.Pigot@anonymised.com<mailto:Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)<mailto:Simon.Pigot@anonymised.commailto:[Simon.Pigot@anonymised.com](mailto:Simon.Pigot@anonymised.com)>>> wrote:
Erk - just looked further down the email and saw that you had already reported open files as deleted ones from the Lucene index - this thread has been going on for too long! (well, too long for my memory anyway! :-)). I’m pretty certain the problem is that which we ran across - I’ll port the patch from BlueNetMEST 1.4.9 to 2.6.4 (which are pretty close) and report back at which time I hope you can test it out. Looks like we’ll need to bring this further forward to 2.8.x and trunk as well.

Cheers and thanks,
Simon

From: Simon.Pigot@anonymised.com [Simon.Pigot@anonymised.com]
Sent: Saturday, 6 October 2012 12:22 AM
To: s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>; fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)><mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>>
Subject: [ExternalEmail] Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Hi Sylvain,

You cannot just upgrade to Lucene 3.6 to fix it (if this is the same problem).

The problem we ran into is as follows:

  • GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
  • IndexReader (LuceneSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
  • When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
  • Lucene is a file based index, rewriting of index deletes some files and creates new ones
  • it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Can you check and see which files are left open? If this is the same problem they will be in the Lucene directory and they will show up as ‘deleted’.

Cheers,
Simon

From: Sylvain GRELLET [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>]
Sent: Friday, 5 October 2012 1:52 AM
To: Pigot, Simon (CMAR, Hobart); fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)><mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)<mailto:fx.prunayre@anonymised.commailto:[fx.prunayre@anonymised.com](mailto:fx.prunayre@anonymised.com)>>
Cc: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>>; Sylvain GRELLET
Subject: Re: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

François, Simon,

A feedback on this aspect after a couple month with a max opened files
raised at 4096 on our production server.
We are still facing the same issue. We have just a bit more time to react.

We are monitoring files opened by Geonetwork on our server.
It appears that the number of files opened by Geonetwork grows steadily
over time (see the 2 images attached).
On “Geonetwork_opened_files.png” you’ll see we had to reload Geonetwork
3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on
Geonetwork_opened_files_third_peak_detail.png), so it seems there is
some kind of garbage collection. But still the trend is not really nice.

@Simon : you mentioned trying Lucene 3.6.
We are on a production server on which some xslts have been modified to
suit our needs.
Is there an simpler solution than moving to the BlueMest branch ? ex :
like changing the Lucene lib/jar used by our Geonetwork and recompile it ?

Cheers
Sylvain

On 29/06/2012 09:41, Sylvain GRELLET wrote:

François, Simon,

Thanks for your answers. I had already raised the max opened files but
not up to 4096 as mentioned in the link provided by François. I’ll give
it a try.

In case this does not work, I’ll try changing the Lucene version.

I’ll let you know.

Sorry for the delay I was at the OGC hydro DWG meeting.

Cheers
Sylvain

On 21/06/2012 02:42, Simon.Pigot@anonymised.com wrote:

Sounds like this could be similar to something Craig Jones and I ran into with Lucene and the IndexReader handling in GeoNetwork - we spent some time on this for a branch of GeoNetwork (BlueNetMEST) and ultimately I ended up having to move to Lucene 3.6 to get it fixed. If it solves your problem then it could be applied to 2.6.x and probably should also be applied to 2.8 and trunk (although trunk/2.8.x are more complicated because of the multi language indexing). It was not applied to these versions as it hadn’t been reported there as a bug and it’s interesting that you hadn’t run into this until recently - anyway you can obviously try it out on 2.6.x to see if it helps.

The branch is BlueNetMEST 1.4.9 and the subversion commits are 9034, 9038, 9039 and 9041 on the BlueNetMEST-1.4.9 sandbox tag (in the GeoNetwork sourceforge subversion repo).

Cheers,
Simon

From: Sylvain Grellet [s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)><mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)<mailto:s.grellet@anonymised.commailto:[s.grellet@anonymised.com](mailto:s.grellet@anonymised.com)>>]
Sent: Thursday, 21 June 2012 1:13 AM
To: geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)><mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)<mailto:geonetwork-devel@lists.sourceforge.netmailto:[geonetwork-devel@lists.sourceforge.net](mailto:geonetwork-devel@lists.sourceforge.net)>>
Subject: [GeoNetwork-devel] GN 2.6.4 / Java 1.7 / Too many open files error

Dear list,

Our Geonetwork 2.6.4 runs under java “1.7.0_04”.

Since recently we have a too many open files error.

It seems there is some kind of garbage collection activity when number
of open files is around 200-300 (number goes up and sometime down)
But since it crosses this “threshold”, it keeps rising until the system
crashes.
What is really surprising is that many of those openfiles are tagged
(deleted) when doing a lsof…

Most of all those numerous files seem to be lucene non spatial indexes.
We tried the “optimize lucene index” in the Admin interfaces but nothing
changed.
We updated java, nothing…

lsof | grep geonetwork output

…
java 23332 root 104r REG 253,0 7271
1902329
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_noo.cfs
java 23332 root 105r REG 253,0 8749
16352071
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nop.cfs
java 23332 root 106r REG 253,0 604391
16352010
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nod.cfs
(deleted)
java 23332 root 107r REG 253,0 10081
1902326
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_no9.cfs
(deleted)
java 23332 root 108r REG 253,0 7471
16352059
/usr/local/apache-tomcat-6.0.26/webapps/geonetwork_CSW/WEB-INF/lucene/nonspatial/_nnz.cfs
(deleted)
…

Could there be a relation with the other issue I raised to the list

today (“CSW / XPath and Logging issue”) ?

Thanks
Sylvain

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)><mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Live Security Virtual Conference
Exclusive live event will cover all the ways today’s security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)><mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)><mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

Don’t let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)><mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)<mailto:GeoNetwork-devel@lists.sourceforge.netmailto:[GeoNetwork-devel@lists.sourceforge.net](mailto:GeoNetwork-devel@lists.sourceforge.net)>>
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork