In the hazelcast module, there is a custom implementation of a jdbcconfig cache provider (HzCacheProvider). The idea is to have shared cache for jdbcconfig between all your nodes. I have found it is a huge drain on resources and it seems to provide no benefits.
- hazelcast distributes catalog events to all nodes, and jdbcconfig invalidates catalog objects based on catalog events; so if each node has its own individual cache that cache is still always kept up to date and catalog changes are distributed immediately to all nodes.
- the hazelcast cache uses XStreamPersister to serialize all the catalog data. that means that each time you take an object out of the cache (so each time you request a catalog object) it loads it from XML and does all the work that is normally only done at start-up (create the data store, try to connect to it, etc...) this is extremely expensive, and on top of that, causes massive traffic between nodes.
we have a configuration with about 800 layers and a health check that is done every 5 minutes that calls a getcapabilities, this was causing 5Mbps traffic between two nodes !!
So I found it is a really good thing to comment the thing out in the spring application context, which stopped all of that traffic and everything is still just as functional. So I wonder, why it was ever created, what was the idea behind it?
Regards
Niels
Nobody answers... does anyone know who developed this? Was it Justin?
Do we still have a maintainer for the hzcluster module?
Regards
Niels
On 12-06-18 11:17, Niels Charlier wrote:
In the hazelcast module, there is a custom implementation of a jdbcconfig cache provider (HzCacheProvider). The idea is to have shared cache for jdbcconfig between all your nodes. I have found it is a huge drain on resources and it seems to provide no benefits.
- hazelcast distributes catalog events to all nodes, and jdbcconfig invalidates catalog objects based on catalog events; so if each node has its own individual cache that cache is still always kept up to date and catalog changes are distributed immediately to all nodes.
- the hazelcast cache uses XStreamPersister to serialize all the catalog data. that means that each time you take an object out of the cache (so each time you request a catalog object) it loads it from XML and does all the work that is normally only done at start-up (create the data store, try to connect to it, etc...) this is extremely expensive, and on top of that, causes massive traffic between nodes.
we have a configuration with about 800 layers and a health check that is done every 5 minutes that calls a getcapabilities, this was causing 5Mbps traffic between two nodes !!
So I found it is a really good thing to comment the thing out in the spring application context, which stopped all of that traffic and everything is still just as functional. So I wonder, why it was ever created, what was the idea behind it?
Regards
Niels
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers... does anyone know who developed this? Was it Justin?
I believe he was... see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea...
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information. == Ing. Andrea Aime @geowolf Technical Lead
GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39
0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549
http://www.geo-solutions.it http://twitter.com/geosolutions_it
------------------------------------------------------- *Con riferimento
alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 -
Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni
circostanza inerente alla presente email (il suo contenuto, gli eventuali
allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i
destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per
errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le
sarei comunque grato se potesse darmene notizia. This email is intended
only for the person or entity to which it is addressed and may contain
information that is privileged, confidential or otherwise protected from
disclosure. We remind that - as provided by European Regulation 2016/679
“GDPR” - copying, dissemination or use of this e-mail or the information
herein by anyone other than the intended recipient is prohibited. If you
have received this email by mistake, please notify us immediately by
telephone or e-mail.*
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
···
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Hello Justin,
Thanks for your answer. I understand that the community status of the module means that it can have some flaws. Not many have contributed to it in years actually. I was just wondering about the intent of the original programmer,if they still remember, so I am better equipped with knowledge to make improvements.
Regards
Niels
···
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Actually, @Steve Ikeoka has made changes to the hazelcast cache in recent times (CC’ed).
I wonder if he has any experience/insight in this matter?
Regards
Niels
···
On 04-07-18 16:03, Niels Charlier wrote:
Hello Justin,
Thanks for your answer. I understand that the community status of the module means that it can have some flaws. Not many have contributed to it in years actually. I was just wondering about the intent of the original programmer,if they still remember, so I am better equipped with knowledge to make improvements.
Regards
Niels
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! [http://sdm.link/slashdot](http://sdm.link/slashdot)
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)
Hey Steve,
Thanks for your response.
Can you give an example of an issue with parent/child objects that could occur in the catalog? I can’t really think of any.
Changes to the catalog always apply to a single object, and avoiding references to old versions of catalog objects is a general issue that is avoided by the use of proxies.
Note also that this cache is only used by jdbcconfig. Jdbcconfig however can also run without clustering, in which case it relies solely on the event system to invalidate the cache when changes happen. So if that system doesn’t work, that wpould mean jdbcconfig can’t work properly without hazelcast, which would be an odd situation and is not really the case.
The use of these resources is all because of this cache, for which I really see no purpose. A cache per node seems like the best idea.
Regards
Niels
···
On 05-07-18 21:53, Ikeoka, Steve wrote:
I agree with you that Hazelcast uses a lot of resources and can really slow down getcapabilities requests but I don’t have any more information about the design of the Hazelcast clustering plugin and I don’t have a cluster environment to test with. If you’ve been able to thoroughly test the Hazelcast clustering plugin without the Hazelcast cache then I would say it’s okay to remove it. One thing that you should test is that there are no issues with parent/child objects when you are relying only on the catalog events to distribute changes across the cluster. I think that that may have been part of the reasoning for the cache serializing/deserializing the catalog data.
Steve Ikeoka
From: Niels Charlier niels@anonymised.com
Sent: Thursday, July 5, 2018 5:16 AM
To: geoserver-devel@lists.sourceforge.net
Cc: Ikeoka, Steve
Subject: Re: [Geoserver-devel] hazelcast/jdbcconfig cache: why?
Actually, @Steve Ikeoka has made changes to the hazelcast cache in recent times (CC’ed).
I wonder if he has any experience/insight in this matter?
Regards
Niels
On 04-07-18 16:03, Niels Charlier wrote:
Hello Justin,
Thanks for your answer. I understand that the community status of the module means that it can have some flaws. Not many have contributed to it in years actually. I was just wondering about the intent of the original programmer,if they still remember, so I am better equipped with knowledge to make improvements.
Regards
Niels
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! [http://sdm.link/slashdot](http://sdm.link/slashdot)
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)
I was just thinking about how catalog objects tend to contain references to other catalog objects (e.g., a layer contains a reference to a resource and one or more styles and a resource contains a reference to a namespace and a store). As long as the Hazelcast catalog events are able to properly propagate changes to catalog objects across the cluster (since each node would have its own instance of the object), then I do think that the Hazelcast cache may be unnecessary.
I don’t see why there would be any issues using jdbcconfig without a cluster.
Steve Ikeoka
···
From: Niels Charlier niels@anonymised.com
Sent: Friday, July 6, 2018 7:21 AM
To: Ikeoka, Steve; Geoserver-devel
Subject: Re: [Geoserver-devel] hazelcast/jdbcconfig cache: why?
Hey Steve,
Thanks for your response.
Can you give an example of an issue with parent/child objects that could occur in the catalog? I can’t really think of any.
Changes to the catalog always apply to a single object, and avoiding references to old versions of catalog objects is a general issue that is avoided by the use of proxies.
Note also that this cache is only used by jdbcconfig. Jdbcconfig however can also run without clustering, in which case it relies solely on the event system to invalidate the cache when changes happen. So if that system doesn’t work, that wpould mean jdbcconfig can’t work properly without hazelcast, which would be an odd situation and is not really the case.
The use of these resources is all because of this cache, for which I really see no purpose. A cache per node seems like the best idea.
Regards
Niels
On 05-07-18 21:53, Ikeoka, Steve wrote:
I agree with you that Hazelcast uses a lot of resources and can really slow down getcapabilities requests but I don’t have any more information about the design of the Hazelcast clustering plugin and I don’t have a cluster environment to test with. If you’ve been able to thoroughly test the Hazelcast clustering plugin without the Hazelcast cache then I would say it’s okay to remove it. One thing that you should test is that there are no issues with parent/child objects when you are relying only on the catalog events to distribute changes across the cluster. I think that that may have been part of the reasoning for the cache serializing/deserializing the catalog data.
Steve Ikeoka
From: Niels Charlier niels@anonymised.com
Sent: Thursday, July 5, 2018 5:16 AM
To: geoserver-devel@lists.sourceforge.net
Cc: Ikeoka, Steve
Subject: Re: [Geoserver-devel] hazelcast/jdbcconfig cache: why?
Actually, @Steve Ikeoka has made changes to the hazelcast cache in recent times (CC’ed).
I wonder if he has any experience/insight in this matter?
Regards
Niels
On 04-07-18 16:03, Niels Charlier wrote:
Hello Justin,
Thanks for your answer. I understand that the community status of the module means that it can have some flaws. Not many have contributed to it in years actually. I was just wondering about the intent of the original programmer,if they still remember, so I am better equipped with knowledge to make improvements.
Regards
Niels
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! [http://sdm.link/slashdot](https://teamworksportal.gdit.com/,DanaInfo=sdm.link+slashdot)
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@anonymised.comsourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://teamworksportal.gdit.com/lists/listinfo/,DanaInfo=lists.sourceforge.net,SSL+geoserver-devel)
Hi Steve,
There is no need to propagate, because all the relationships between catalog objects are done via resolvingproxies that use the catalog and the id to look up the latest version of the object.
For example, if an object belongs to a workspace and you modify that workspace, the resolvingproxy will find the new version using the ID and the catalog.
Regards
Niels
···
On 06-07-18 17:04, Ikeoka, Steve wrote:
I was just thinking about how catalog objects tend to contain references to other catalog objects (e.g., a layer contains a reference to a resource and one or more styles and a resource contains a reference to a namespace and a store). As long as the Hazelcast catalog events are able to properly propagate changes to catalog objects across the cluster (since each node would have its own instance of the object), then I do think that the Hazelcast cache may be unnecessary.
I don’t see why there would be any issues using jdbcconfig without a cluster.
Steve Ikeoka
From: Niels Charlier niels@anonymised.com
Sent: Friday, July 6, 2018 7:21 AM
To: Ikeoka, Steve; Geoserver-devel
Subject: Re: [Geoserver-devel] hazelcast/jdbcconfig cache: why?
Hey Steve,
Thanks for your response.
Can you give an example of an issue with parent/child objects that could occur in the catalog? I can’t really think of any.
Changes to the catalog always apply to a single object, and avoiding references to old versions of catalog objects is a general issue that is avoided by the use of proxies.
Note also that this cache is only used by jdbcconfig. Jdbcconfig however can also run without clustering, in which case it relies solely on the event system to invalidate the cache when changes happen. So if that system doesn’t work, that wpould mean jdbcconfig can’t work properly without hazelcast, which would be an odd situation and is not really the case.
The use of these resources is all because of this cache, for which I really see no purpose. A cache per node seems like the best idea.
Regards
Niels
On 05-07-18 21:53, Ikeoka, Steve wrote:
I agree with you that Hazelcast uses a lot of resources and can really slow down getcapabilities requests but I don’t have any more information about the design of the Hazelcast clustering plugin and I don’t have a cluster environment to test with. If you’ve been able to thoroughly test the Hazelcast clustering plugin without the Hazelcast cache then I would say it’s okay to remove it. One thing that you should test is that there are no issues with parent/child objects when you are relying only on the catalog events to distribute changes across the cluster. I think that that may have been part of the reasoning for the cache serializing/deserializing the catalog data.
Steve Ikeoka
From: Niels Charlier niels@anonymised.com
Sent: Thursday, July 5, 2018 5:16 AM
To: geoserver-devel@lists.sourceforge.net
Cc: Ikeoka, Steve
Subject: Re: [Geoserver-devel] hazelcast/jdbcconfig cache: why?
Actually, @Steve Ikeoka has made changes to the hazelcast cache in recent times (CC’ed).
I wonder if he has any experience/insight in this matter?
Regards
Niels
On 04-07-18 16:03, Niels Charlier wrote:
Hello Justin,
Thanks for your answer. I understand that the community status of the module means that it can have some flaws. Not many have contributed to it in years actually. I was just wondering about the intent of the original programmer,if they still remember, so I am better equipped with knowledge to make improvements.
Regards
Niels
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! [http://sdm.link/slashdot](https://teamworksportal.gdit.com/,DanaInfo=sdm.link+slashdot)
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://teamworksportal.gdit.com/lists/listinfo/,DanaInfo=lists.sourceforge.net,SSL+geoserver-devel)
Justin,
Related question, do you by any chance remember why you made a “reload synchroniser” and made it the default instead of “event synchroniser” which is much more efficient and generates far less traffic. Furthermore, I found that if you generate a whole lot of events after each other (causing reloads in all the other nodes) there is a chance at some point you will get a deadlock in the cache and your node becomes malfunctioning. The issue is however very hard to replicate or to debug.
Regards
Niels
···
On 04-07-18 15:16, Justin Deoliveira wrote:
I haven’t committed to that module in many years so I am the wrong person to ask. I would start by looking at someone who more recently has made changes to it, someone from Boundless perhaps. I will say when I worked on it it was a prototype, something that hadn’t seen any use in production, so I am not surprised if it has issues like this. Not an uncommon occurrence for community modules I would say. Given no-one else has responded it seems safe for you to make whatever changes to it you see fit.
On Wed, Jul 4, 2018 at 6:06 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
I can’t recall anything specific sorry. When I worked on that code it was very much a prototype that I never put through any sort of stress or scale testing. I suspect I used that because it “just worked” in the development environment I was working in.
···
On Wed, Jul 4, 2018 at 1:37 PM, Niels Charlier <niels@anonymised.com> wrote:
Nobody answers… does anyone know who developed this? Was it Justin?
I believe he was… see first commit in both these files:
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/src/main/java/org/geoserver/cluster/ClusterConfig.java
https://github.com/geoserver/geoserver/commits/master/src/community/hz-cluster/pom.xml
Do we still have a maintainer for the hzcluster module?
No idea…
Regards
Andrea
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.