[GeoNetwork-users] I-O Exception (Too many open files)

Hi list,

I am new in the list and I am facing a problem which I will try to
summarize:

We are running on a Linux server a GN version 2.0.2 that has been working
pretty fine during the last months.

Recently, the following events have started to happen:

- Almost at the end of each day GN is having an I-O Exception when
dispatching to output with the stylesheet "search-results.xsl". To be more
precise, it's not this xsl that raises the problem but one of those
following this one in the process of showing a request result
(metadata-iso19115.xsl, metadata-fgdc-std.xsl , main.xsl or metadata.xsl,
etc (see Example 1 below, sorry logs are in French :wink: ).

- Usually, once it happens for the 1st time of the day it is
quickly followed by other errors having the same symptoms. As a result GN
his having, after a while, another kind of error:
"java.io.FileNotFoundException:
/home/www/www.sandre.eaufrance.fr.geonetwork/web/loc/fr/xml/strings.xml (Too
many open files)" (see Example 2 below).

- And, finally, we've got a "java.net.SocketException: Too many
open files" that causes our Tomcat server to freeze (see Example 3 below),

- We restart the server and everything works fine during the whole
day until the next I-O Exception !!

I have looked through the previous forum threads and haven't found any
solution.

Does someone have an idea that could help us go forward . :slight_smile: ?

Here are the pieces of reasoning I have made:

- The request sent to GN by our users and causing the failure seems not to
be the same. I have identified no specific pattern except that they only use
'any' and 'selregion' parameters) but they often work fine during the day ->
problem due to system settings ?

- We have not changed majors settings on our server (nor on GN),

- Things are not always happening at the same hour -> not related to a cron
task.

As a result I have several questions:

- How can we trigger a request only having 'any' and 'selregion'
parameters sent to GN? Even in simple search, I've got other search
parameters appearing. It could help me trying to reproduce the error?

- The I-O Exception causes this cascade of problems (cause =
Concurrent access to an xsl ? or .?) but apparently xml files are not closed
when this I-O Exception happens which causes that "Too many open files"
error -> is that normal ?

- What can cause this I-O Exception (of course)?

*************

Example 1 :

**************

2008-03-09 07:24:31 | HTML Request (GET) : /geonetwork/srv/fr/main.search

2008-03-09 07:24:31 | Session created for client :192.168.1.105

2008-03-09 07:24:31 | Dispatching : main.search

2008-03-09 07:24:31 | -> parameters are :

2008-03-09 07:24:31 | any = var

2008-03-09 07:24:31 | selregion =
9999;-61.79611;-61.18708;15.87;16.51292

2008-03-09 07:24:31 | [main.search] Creating default search data

2008-03-09 07:24:31 | [main.search] Creating searchers

2008-03-09 07:24:31 | [main.search] Getting summary

2008-03-09 07:24:31 | -> dispatching to output for : main.search

2008-03-09 07:24:31 | -> forwarding to : main.present

2008-03-09 07:24:31 | Dispatching : main.present

2008-03-09 07:24:31 | -> parameters are :

2008-03-09 07:24:31 | summary =

2008-03-09 07:24:31 | forward =

2008-03-09 07:24:32 | -> dispatching to output for : main.present

2008-03-09 07:24:32 | -> transforming with stylesheet :
/home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-results.xsl

file:///home/www/xxx/web/xsl/search-results.xsl; Ligne #6; Colonne #32;
Exception d'E-S avec le fichier de la feuille de style : main.xsl

file:///home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-results.x
sl; Ligne #7; Colonne #36; Exception d'E-S avec le fichier de la feuille de
style : metadata.xsl

file:///home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-results.x
sl; Ligne #25; Colonne #40; javax.xml.transform.TransformerException: Erreur
de ElemTemplateElement : formTitle

*************

Example 2 :

**************

2008-03-09 07:25:33 | HTML Request (GET) : /geonetwork/srv/fr/main.search

2008-03-09 07:25:33 | Session created for client :192.168.1.105

2008-03-09 07:25:33 | Dispatching : main.search

2008-03-09 07:25:33 | -> parameters are :

2008-03-09 07:25:33 | any = bassins

2008-03-09 07:25:33 | selregion =
68;-5.79028;9.56222;41.36493;51.09111

2008-03-09 07:25:33 | [main.search] Creating default search data

2008-03-09 07:25:33 | [main.search] Creating searchers

reconnecting: 3628448>=3600000 ms since last connection

2008-03-09 07:25:33 | [main.search] Getting summary

2008-03-09 07:25:33 | -> dispatching to output for : main.search

2008-03-09 07:25:33 | -> forwarding to : main.present

2008-03-09 07:25:33 | Dispatching : main.present

2008-03-09 07:25:33 | -> parameters are :

2008-03-09 07:25:33 | summary =

2008-03-09 07:25:33 | forward =

2008-03-09 07:25:33 | -> dispatching to output for : main.present

2008-03-09 07:25:33 | [main.present] Exception executing gui service :
java.io.FileNotFoundException: /home/www/www.sandre.ea

ufrance.fr.geonetwork/web/loc/fr/xml/strings.xml (Too many open files)

2008-03-09 07:25:33 | [main.present] (C) Stack trace is :

java.io.FileNotFoundException:
/home/www/www.sandre.eaufrance.fr.geonetwork/web/loc/fr/xml/strings.xml (Too
many open files)

*************

Example 3 :

*************

java.net.SocketException: Too many open files

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

        at java.net.ServerSocket.implAccept(ServerSocket.java:450)

        at java.net.ServerSocket.accept(ServerSocket.java:421)

        at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultSe
rverSocketFactory.java:60)

        at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java
:407)

        at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWo
rkerThread.java:70)

        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)

        at java.lang.Thread.run(Thread.java:595)

Thank you in advance for your help

Sylvain

Hi Sylvain,

The problem seems to be due to a linux kernel limit.
You can check with this command : "ulimit -a" the value of the field "open
files"

Usualy on the linux system the maximum number of opened files is set to 1024
(per process)

If you increase this value then I think your problem will be fixed.

Regards,

Renaud

java.net.SocketException: Too many open files

On Fri, Mar 14, 2008 at 3:05 PM, Sylvain Grellet <s.grellet@anonymised.com> wrote:

Hi list,

I am new in the list and I am facing a problem which I will try to
summarize:

We are running on a Linux server a GN version 2.0.2 that has been working
pretty fine during the last months.

Recently, the following events have started to happen:

- Almost at the end of each day GN is having an I-O Exception
when
dispatching to output with the stylesheet "search-results.xsl". To be more
precise, it's not this xsl that raises the problem but one of those
following this one in the process of showing a request result
(metadata-iso19115.xsl, metadata-fgdc-std.xsl , main.xsl or metadata.xsl,
etc (see Example 1 below, sorry logs are in French :wink: ).

- Usually, once it happens for the 1st time of the day it is
quickly followed by other errors having the same symptoms. As a result GN
his having, after a while, another kind of error:
"java.io.FileNotFoundException:
/home/www/www.sandre.eaufrance.fr.geonetwork/web/loc/fr/xml/strings.xml
(Too
many open files)" (see Example 2 below).

- And, finally, we've got a "java.net.SocketException: Too many
open files" that causes our Tomcat server to freeze (see Example 3 below),

- We restart the server and everything works fine during the
whole
day until the next I-O Exception !!

I have looked through the previous forum threads and haven't found any
solution.

Does someone have an idea that could help us go forward . :slight_smile: ?

Here are the pieces of reasoning I have made:

- The request sent to GN by our users and causing the failure seems not to
be the same. I have identified no specific pattern except that they only
use
'any' and 'selregion' parameters) but they often work fine during the day
->
problem due to system settings ?

- We have not changed majors settings on our server (nor on GN),

- Things are not always happening at the same hour -> not related to a
cron
task.

As a result I have several questions:

- How can we trigger a request only having 'any' and 'selregion'
parameters sent to GN? Even in simple search, I've got other search
parameters appearing. It could help me trying to reproduce the error?

- The I-O Exception causes this cascade of problems (cause =
Concurrent access to an xsl ? or .?) but apparently xml files are not
closed
when this I-O Exception happens which causes that "Too many open files"
error -> is that normal ?

- What can cause this I-O Exception (of course)?

*************

Example 1 :

**************

2008-03-09 07:24:31 | HTML Request (GET) : /geonetwork/srv/fr/main.search

2008-03-09 07:24:31 | Session created for client :192.168.1.105

2008-03-09 07:24:31 | Dispatching : main.search

2008-03-09 07:24:31 | -> parameters are :

2008-03-09 07:24:31 | any = var

2008-03-09 07:24:31 | selregion =
9999;-61.79611;-61.18708;15.87;16.51292

2008-03-09 07:24:31 | [main.search] Creating default search data

2008-03-09 07:24:31 | [main.search] Creating searchers

2008-03-09 07:24:31 | [main.search] Getting summary

2008-03-09 07:24:31 | -> dispatching to output for : main.search

2008-03-09 07:24:31 | -> forwarding to : main.present

2008-03-09 07:24:31 | Dispatching : main.present

2008-03-09 07:24:31 | -> parameters are :

2008-03-09 07:24:31 | summary =

2008-03-09 07:24:31 | forward =

2008-03-09 07:24:32 | -> dispatching to output for : main.present

2008-03-09 07:24:32 | -> transforming with stylesheet :
/home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-results.xsl

file:///home/www/xxx/web/xsl/search-results.xsl; Ligne #6; Colonne #32;
Exception d'E-S avec le fichier de la feuille de style : main.xsl

file:///home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-
results.x
sl; Ligne #7; Colonne #36; Exception d'E-S avec le fichier de la feuille
de
style : metadata.xsl

file:///home/www/www.sandre.eaufrance.fr.geonetwork/web/xsl/search-
results.x
sl; Ligne #25; Colonne #40; javax.xml.transform.TransformerException:
Erreur
de ElemTemplateElement : formTitle

*************

Example 2 :

**************

2008-03-09 07:25:33 | HTML Request (GET) : /geonetwork/srv/fr/main.search

2008-03-09 07:25:33 | Session created for client :192.168.1.105

2008-03-09 07:25:33 | Dispatching : main.search

2008-03-09 07:25:33 | -> parameters are :

2008-03-09 07:25:33 | any = bassins

2008-03-09 07:25:33 | selregion =
68;-5.79028;9.56222;41.36493;51.09111

2008-03-09 07:25:33 | [main.search] Creating default search data

2008-03-09 07:25:33 | [main.search] Creating searchers

reconnecting: 3628448>=3600000 ms since last connection

2008-03-09 07:25:33 | [main.search] Getting summary

2008-03-09 07:25:33 | -> dispatching to output for : main.search

2008-03-09 07:25:33 | -> forwarding to : main.present

2008-03-09 07:25:33 | Dispatching : main.present

2008-03-09 07:25:33 | -> parameters are :

2008-03-09 07:25:33 | summary =

2008-03-09 07:25:33 | forward =

2008-03-09 07:25:33 | -> dispatching to output for : main.present

2008-03-09 07:25:33 | [main.present] Exception executing gui service :
java.io.FileNotFoundException: /home/www/www.sandre.ea

ufrance.fr.geonetwork/web/loc/fr/xml/strings.xml (Too many open files)

2008-03-09 07:25:33 | [main.present] (C) Stack trace is :

java.io.FileNotFoundException:
/home/www/www.sandre.eaufrance.fr.geonetwork/web/loc/fr/xml/strings.xml
(Too
many open files)

*************

Example 3 :

*************

java.net.SocketException: Too many open files

       at java.net.PlainSocketImpl.socketAccept(Native Method)

       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

       at java.net.ServerSocket.implAccept(ServerSocket.java:450)

       at java.net.ServerSocket.accept(ServerSocket.java:421)

       at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket
(DefaultSe
rverSocketFactory.java:60)

       at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(
PoolTcpEndpoint.java
:407)

       at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt
(LeaderFollowerWo
rkerThread.java:70)

       at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.jav
a:684)

       at java.lang.Thread.run(Thread.java:595)

Thank you in advance for your help

Sylvain

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork

Hi Sylvain,

Sylvain Grellet wrote:

I am new in the list :

Welcome !

- How can we trigger a request only having 'any' and 'selregion'
parameters sent to GN?

http://mygeonetwork/geonetwork/srv/en/main.search?any=var&selregion=blabal
will launch the request. So you could reproduce the problem with the
same parameters - if the problem is linked to the param and not to the
system.

Is the problem could be linked to somebody harvesting your node using
another app (geonetwork) ?

2008-03-09 07:24:31 | -> parameters are :
2008-03-09 07:24:31 | any = var
2008-03-09 07:24:31 | selregion =
9999;-61.79611;-61.18708;15.87;16.51292

Do you have always the same search parameters ? If yes, it could be
somebody in the south of France near the "var" :wink:

Ciao. Francois

Hy guys,

- I have tried to access the main.search with the content of a request that
had triggered a problem -> it has not raised a thing ;(
- Concerning the openfiles limit, I am afraid I will not solve the cause but
only the delay the consequences. Obviously, there is a program that does not
clause its open files. But that gave me the idea to do a little "lsof -r |
grep Geonetwork" and redirect it to a log file to find the guilty program.

François, the request I tried was :
2008-03-11 06:25:25 | any = dehors
2008-03-11 06:25:25 | selregion =
87;-61.79611;-61.18708;15.87;16.51292
This guy is not only in the "var" he is ... everywhere outside !

Sylvain

****************************************************************************
Sylvain GRELLET
OIE/IOW (Office International de l'Eau/International Office for Water)
Ingénieur SIG
Centre National d'Information et de Documentation sur l'Eau

15 rue Edouard Chamberland
87065 LIMOGES Cedex, France
Tel. 33.5.55.11.47.95
Fax 33.5.55.11.47.48
Web: http://www.oieau.org
****************************************************************************

-----Message d'origine-----
De : Francois-Xavier Prunayre
[mailto:francois-xavier.prunayre@anonymised.com]
Envoyé : vendredi 14 mars 2008 15:26
À : Sylvain Grellet
Cc : geonetwork-users@lists.sourceforge.net
Objet : Re: [GeoNetwork-users] I-O Exception (Too many open files)

Hi Sylvain,

Sylvain Grellet wrote:

I am new in the list :

Welcome !

- How can we trigger a request only having 'any' and 'selregion'
parameters sent to GN?

http://mygeonetwork/geonetwork/srv/en/main.search?any=var&selregion=blabal
will launch the request. So you could reproduce the problem with the
same parameters - if the problem is linked to the param and not to the
system.

Is the problem could be linked to somebody harvesting your node using
another app (geonetwork) ?

2008-03-09 07:24:31 | -> parameters are :
2008-03-09 07:24:31 | any = var
2008-03-09 07:24:31 | selregion =
9999;-61.79611;-61.18708;15.87;16.51292

Do you have always the same search parameters ? If yes, it could be
somebody in the south of France near the "var" :wink:

Ciao. Francois

Additionnal info :

- I have taken a look at the result given by lsof and I am surprise to see
that those 2 documents are open several times simultaneously :
"WEB-INF/lucene/_t8.cfs" and "WEB-INF/lucene/_sy.cfs"
Is that a normal behaviour? I will trace this during the WE.

- Regarding the fact of being harvested by another app (geonetwork). How can
one know that one is being harvested?

Thanks
Sylvain

-----Message d'origine-----
De : geonetwork-users-bounces@lists.sourceforge.net
[mailto:geonetwork-users-bounces@lists.sourceforge.net] De la part de
Sylvain Grellet
Envoyé : vendredi 14 mars 2008 15:45
À : geonetwork-users@lists.sourceforge.net
Objet : Re: [GeoNetwork-users] I-O Exception (Too many open files)

Hy guys,

- I have tried to access the main.search with the content of a request that
had triggered a problem -> it has not raised a thing ;(
- Concerning the openfiles limit, I am afraid I will not solve the cause but
only the delay the consequences. Obviously, there is a program that does not
clause its open files. But that gave me the idea to do a little "lsof -r |
grep Geonetwork" and redirect it to a log file to find the guilty program.

François, the request I tried was :
2008-03-11 06:25:25 | any = dehors
2008-03-11 06:25:25 | selregion =
87;-61.79611;-61.18708;15.87;16.51292
This guy is not only in the "var" he is ... everywhere outside !

Sylvain

****************************************************************************
Sylvain GRELLET
OIE/IOW (Office International de l'Eau/International Office for Water)
Ingénieur SIG
Centre National d'Information et de Documentation sur l'Eau

15 rue Edouard Chamberland
87065 LIMOGES Cedex, France
Tel. 33.5.55.11.47.95
Fax 33.5.55.11.47.48
Web: http://www.oieau.org
****************************************************************************

-----Message d'origine-----
De : Francois-Xavier Prunayre
[mailto:francois-xavier.prunayre@anonymised.com]
Envoyé : vendredi 14 mars 2008 15:26
À : Sylvain Grellet
Cc : geonetwork-users@lists.sourceforge.net
Objet : Re: [GeoNetwork-users] I-O Exception (Too many open files)

Hi Sylvain,

Sylvain Grellet wrote:

I am new in the list :

Welcome !

- How can we trigger a request only having 'any' and 'selregion'
parameters sent to GN?

http://mygeonetwork/geonetwork/srv/en/main.search?any=var&selregion=blabal
will launch the request. So you could reproduce the problem with the
same parameters - if the problem is linked to the param and not to the
system.

Is the problem could be linked to somebody harvesting your node using
another app (geonetwork) ?

2008-03-09 07:24:31 | -> parameters are :
2008-03-09 07:24:31 | any = var
2008-03-09 07:24:31 | selregion =
9999;-61.79611;-61.18708;15.87;16.51292

Do you have always the same search parameters ? If yes, it could be
somebody in the south of France near the "var" :wink:

Ciao. Francois

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at
http://sourceforge.net/projects/geonetwork