Hi,
We’re tying to harvest large GN catalogs with more than 20 000 records (i.e. http://geomatics.nlr.nl/unsdi). At the beginning, everything is fine but after a few hours and about 10 000 metadata harvested, the catalog raises an error "GC overhead limit exceeded ».
We made lots of test, harvesting other catalogs, with the same result.
Our configuration : GN 3.0.4 - 12 Go RAM (-Xmx 8192m) - 8 cores - Tomcat 8 - JDK 8.0_91
Currently, catalogs contains about 30 thousands metadata. We use PostGIS as spatial index. Complete indexation takes about 10 minutes.
During harvesting, we observed that memory usage grew steadily during one ou 2 hours (from 4 Go until about 9 Go). Then harvesting starts to become slower for a time, long or short. And then it crashes, often with the "GC overhead limit exceeded" error.
However, when metadata are already in our catalog from a previous harvesting, we don’t encounter the problem. This way we’re able to harvest a catalog with 20 000 metadata in 2 or 3 times (restarting tomcat after each crash) and then, the full harvesting of this catalog is ok.
Have you ever meet that issue ? Can you help us ?
Thanks in advance
Matthieu Lefort
Hi Matthieu
2016-07-19 10:23 GMT+02:00 Matthieu Lefort <m.lefort@anonymised.com>:
Hi,
We’re tying to harvest large GN catalogs with more than 20 000 records
(i.e. http://geomatics.nlr.nl/unsdi). At the beginning, everything is
fine but after a few hours and about 10 000 metadata harvested, the catalog
raises an error "GC overhead limit exceeded ».
We made lots of test, harvesting other catalogs, with the same result.
Our configuration : GN 3.0.4 - 12 Go RAM (-Xmx 8192m) - 8 cores - Tomcat 8
- JDK 8.0_91
Currently, catalogs contains about 30 thousands metadata. We use PostGIS
as spatial index. Complete indexation takes about 10 minutes.
During harvesting, we observed that memory usage grew steadily during one
ou 2 hours (from 4 Go until about 9 Go). Then harvesting starts to become
slower for a time, long or short. And then it crashes, often with the "GC
overhead limit exceeded" error.
However, when metadata are already in our catalog from a previous
harvesting, we don’t encounter the problem. This way we’re able to harvest
a catalog with 20 000 metadata in 2 or 3 times (restarting tomcat after
each crash) and then, the full harvesting of this catalog is ok.
Have you ever meet that issue ? Can you help us ?
Testing on develop branch Postgres with Xmx2g after harvesting 32K records
(ending at 3h45), memory is ok.
!image.png|355x279
I don't think there is much difference with 3.0.4 on this. So no suggestion.
BTW, if you need better performance, you may investigate the following:
* by default harvesting execution is not multi-thread. Only parallel
harvester executions is using a thread pool. Quite easily indexing could be
multi-thread. eg LFS (local file system) harvester is inserting records and
at the end index everything.
* CSW is making GetRecords and then GetRecordsById, opting for a strategy
only based on GetRecords operation and indexing page per page would be
better (that what is done in the daobs project
https://github.com/INSPIRE-MIF/daobs and is quite fast to harvest all EU
member states CSWs)
Some numbers on the work done on LFS last year for loading 230K records:
* H2 : 20K records in 2 hours and slowing down
* PostGIS: 16 hours
* PostGIS with indexing on 4 threads: 6 hours.
So there is room for improvements in CSW and Geonet harvesters.
Francois
Thanks in advance
Matthieu Lefort
Hi François,
Thanks for the reply.
On your graph, there is a sharp drop every 25 min. (i.e. at 2:15 pm) that enables not to exceed heap size. That seems not to be happening during my tests.
I tried to use -XX:+UseParallelOldGC in JAVA options. It appeared to be better. Nevertheless, I couldn’t harvest 30k records at once.
Matthieu
Le 28 juil. 2016 à 07:21, Francois Prunayre <fx.prunayre@anonymised.com> a écrit :
Hi Matthieu
2016-07-19 10:23 GMT+02:00 Matthieu Lefort <m.lefort@anonymised.com <mailto:m.lefort@anonymised.com>>:
Hi,
We’re tying to harvest large GN catalogs with more than 20 000 records (i.e. http://geomatics.nlr.nl/unsdi). At the beginning, everything is fine but after a few hours and about 10 000 metadata harvested, the catalog raises an error "GC overhead limit exceeded ».
We made lots of test, harvesting other catalogs, with the same result.
Our configuration : GN 3.0.4 - 12 Go RAM (-Xmx 8192m) - 8 cores - Tomcat 8 - JDK 8.0_91
Currently, catalogs contains about 30 thousands metadata. We use PostGIS as spatial index. Complete indexation takes about 10 minutes.
During harvesting, we observed that memory usage grew steadily during one ou 2 hours (from 4 Go until about 9 Go). Then harvesting starts to become slower for a time, long or short. And then it crashes, often with the "GC overhead limit exceeded" error.
However, when metadata are already in our catalog from a previous harvesting, we don’t encounter the problem. This way we’re able to harvest a catalog with 20 000 metadata in 2 or 3 times (restarting tomcat after each crash) and then, the full harvesting of this catalog is ok.
Have you ever meet that issue ? Can you help us ?
Testing on develop branch Postgres with Xmx2g after harvesting 32K records (ending at 3h45), memory is ok.
<image.png>
I don't think there is much difference with 3.0.4 on this. So no suggestion.
BTW, if you need better performance, you may investigate the following:
* by default harvesting execution is not multi-thread. Only parallel harvester executions is using a thread pool. Quite easily indexing could be multi-thread. eg LFS (local file system) harvester is inserting records and at the end index everything.
* CSW is making GetRecords and then GetRecordsById, opting for a strategy only based on GetRecords operation and indexing page per page would be better (that what is done in the daobs project https://github.com/INSPIRE-MIF/daobs and is quite fast to harvest all EU member states CSWs)
Some numbers on the work done on LFS last year for loading 230K records:
* H2 : 20K records in 2 hours and slowing down
* PostGIS: 16 hours
* PostGIS with indexing on 4 threads: 6 hours.
So there is room for improvements in CSW and Geonet harvesters.
Francois
Thanks in advance
Matthieu Lefort
