[Geoserver-devel] FileSystemWatcher nightmare

Hi there,

Got a situation here: data directory with 100 workspaces and over 80K layers. Gotta be used in a clustering setup with the JMS clustering extension.

Good news is 2.16.0, with the latest startup fixes, starts up in 40 seconds as opposed to 8-9 minutes for 2.15.0. Haven’t tried 2.15.3 but the fixes are probably there too.

Bad news is FileSystemWatcher is killing the machine. For each instance it gets one thread eating 100% of one CPU all the time. Reported this before, just sharing my pain.

Been looking at replacing the internals of FileSystemWatcher to use java.nio.file.WatchService, and although at a first glance it looks pretty straightforward, it’s not gonna be that simple.

There are a number of situations where it just won’t work, basically when the data dir is on any sort of mapped filesystem (NFS, SAMBA, Docker bind mounts), for which the Linux implementation doesn’t work as it relies on inotify.

The MacOS implementation is a polling one, but it’s gonna be better than the current custom one anyways. And for Windows, I don’t know yet.

In any case it’s clear that anyone that has a large catalog will most probably have it on a shared filesystem to be used by several instances.

So this might require a GSIP, which I’m willing to start, in order to tackle on the different scenarios, as it can get even more involved as above described.

PS: if you’re gonna mention jdbccatalog don’t even bother. Having a better catalog backend is a totally (and worthy) different topic.

Cheers,

···

Gabriel Roldán

Bad news is FileSystemWatcher is killing the machine. For each instance it gets one thread eating 100% of one CPU all the time. Reported this before, just sharing my pain.

Been looking at replacing the internals of FileSystemWatcher to use java.nio.file.WatchService, and although at a first glance it looks pretty straightforward, it’s not gonna be that simple.

Darn, that was the point of making the interface (since we could not use WatchService on Java 7 yet).

There are a number of situations where it just won’t work, basically when the data dir is on any sort of mapped filesystem (NFS, SAMBA, Docker bind mounts), for which the Linux implementation doesn’t work as it relies on inotify.

The MacOS implementation is a polling one, but it’s gonna be better than the current custom one anyways. And for Windows, I don’t know yet.

Still this was only supposed to be used for a few files like the security settings which require active notification? If a watcher is being setup beyond that … is it a mistake?

In any case it’s clear that anyone that has a large catalog will most probably have it on a shared filesystem to be used by several instances.
So this might require a GSIP, which I’m willing to start, in order to tackle on the different scenarios, as it can get even more involved as above described.

Before you go that far can we talk about how FileSystemWatcher is being used…


Jody

Bad news is FileSystemWatcher is killing the machine. For each instance it gets one thread eating 100% of one CPU all the time. Reported this before, just sharing my pain.

Been looking at replacing the internals of FileSystemWatcher to use java.nio.file.WatchService, and although at a first glance it looks pretty straightforward, it’s not gonna be that simple.

Darn, that was the point of making the interface (since we could not use WatchService on Java 7 yet).

There are a number of situations where it just won’t work, basically when the data dir is on any sort of mapped filesystem (NFS, SAMBA, Docker bind mounts), for which the Linux implementation doesn’t work as it relies on inotify.

The MacOS implementation is a polling one, but it’s gonna be better than the current custom one anyways. And for Windows, I don’t know yet.

Still this was only supposed to be used for a few files like the security settings which require active notification? If a watcher is being setup beyond that … is it a mistake?

In any case it’s clear that anyone that has a large catalog will most probably have it on a shared filesystem to be used by several instances.
So this might require a GSIP, which I’m willing to start, in order to tackle on the different scenarios, as it can get even more involved as above described.

Before you go that far can we talk about how FileSystemWatcher is being used…

Right. It’s DefaultTileLayerCatalog listening to changes to gwc-layers/* in order to update tile layer configs when they’re updated externally on a cluster (added at commit 966a8af90fc6b68725a78b09829c3cbc7360242e).

For the time being, I managed to reduce the polling runtime from over 2 minutes to ~350 milliseconds, the CPU usage from constant 100% to peaks bellow 25% (or so says htop). Memory usage: old gen up to 477MB from 450MB, eden space peaks and down quickly of 1GB instead of fixed at around 850MB.

Got some clean up to do before issueing a PR, but here it is:
https://github.com/groldan/geoserver/commit/0ba1614a7f9b1b2bea5647160441f3bd534c9979

Gabriel

···

Gabriel Roldán

That is … unexpected.

So there are other solutions to notify tile cache state across the cluster right. Are they more appropriate then depending on file system polling here?

···


Jody Garnett

That is … unexpected.

So there are other solutions to notify tile cache state across the cluster right. Are they more appropriate then depending on file system polling here?

I would suspect so. The catalog does so it should be possible to do the same with the tile layer catalog. Meanwhile I wouldn’t mind this improvement.

···

Gabriel Roldán

Thinking further … listening to “gwc-layers/*” is a poor idea and unnecessary.

Since we completely know the names of these files (from their matching layer objects). There should be no need to scan the directory for changes … we have enough information to do this file by file which should be more efficient when using opening system watcher.

Is it possible to detect when we are only of these mapped file systems in order to warn the user that a different approach will be required to keep nodes in the cluster advised of changes?

···


Jody Garnett

Similar to how it’s done for the PropertyFileWatcher (adds a small delay between two consequent checks to avoid slamming the file system with
high rate checks, which would be likely when working on cached tiles)

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

So how is this one progressing? Seems stuck

Cheers
Andrea

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hey,

not stuck, just waiting for free time to spend on it.
Got a PR btw: https://github.com/geoserver/geoserver/pull/3860

Note this addresses the issue itself, does not:

  • implement the switch to using WatchService, since that can be more involved and require proper testing under different scenarios to be really confident about it.
  • have an opinion on whether listening to gwc-layers change events is good or bad. Changing that would require further analysis and a different jira ticket. If we settle on a prescribed clustering strategy, say, JMS clustering extension, it should implement the necessary message handling for listening to gwc-layers to not be necessary, if it doesn’t already.

Cheers,
Gabriel

···

Gabriel Roldán