hello lists,
our custom GeoNetwork installation sometimes stops working correctly, with
messages in the logs about "Too many open files".
Running on Ubuntu with Tomcat. Also Apache and Postgres are involved. The
machine has 8G RAM.
The default limit of open files is 1024. Having set it to 16000 seems to
have solved, or alleviated, the problem.
Does anyone have a recommendation about what is the ideal limit to set?
Kind regards
Heikki Doeleman
Hello Heikki,
I had this problem recently with an instance of the BlueNetMEST.
When I looked into it, the problem was caused by not closing the lucene
IndexReader. Every time metadata is changed and the metadata is
reindexed, this changes the lucene index files (creating new
ones/modifying old ones). The next time a search is done, the new files
are opened, and the old ones are not closed.
The more your data changes and the more searches that are done while
this is happening the more quickly the open files increase (I was
running into the 1024 limit every 3rd day mostly because I'm using a
file system harvester nightly that changes the lucene index rather than
because of manually entered metadata).
I applied the attached patch to address the issue last week in my
instance. The open files now don't creep up after performing a harvest
and doing a search which is what I was looking for. Its still early
days, but I haven't seen any side affects yet.
I based my change on previous changes made to close the IndexSearcher
but had to modify it when the number of open files still crept up.
My limit is still set at 1024 although from what I've read while reading
up on this topic - for a dedicated server the limit certainly can be
increased well above this. 1024 is a default meant for a machine with
many users.
Note that this code has changed now in trunk with Simon's performance
enhancements and this now does not look to be an issue there.
Regards,
--
Craig Jones
eMII Infrastructure Programmer
IMOS e-Marine Information Infrastructure Facility (eMII)
Ph: +61 3 6226 8567
On Thu, 2010-04-15 at 13:32 +0200, heikki wrote:
hello lists,
our custom GeoNetwork installation sometimes stops working correctly, with
messages in the logs about "Too many open files".Running on Ubuntu with Tomcat. Also Apache and Postgres are involved. The
machine has 8G RAM.The default limit of open files is 1024. Having set it to 16000 seems to
have solved, or alleviated, the problem.Does anyone have a recommendation about what is the ideal limit to set?
Kind regards
Heikki Doeleman
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
GeoNetwork-users mailing list
GeoNetwork-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
too-many-open-files.patch (7.79 KB)