[GeoNetwork-devel] Questions about multi-node geonetwork 3.4

GeoNetwork GeoNetwork Developer
1

Hi List,

I have a few questions about running geonetwork 3.4.x in multi-node mode. Is it possible to use elasticsearch and kibana in this mode- I’m worried that there will be one single index and dashboard so all administrators will see statistics for all nodes.

Secondly, when I make changes (for example to a schema) is there a way of automatically propagating those changes to all nodes, or do I have to do it manually to each one?

Many thanks

Jo

···

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

2

Hi Jo and all,

About your questions:

  • current elasticsearch (ES) index config is bound to the webapplication and there is no way to define a specific ES index on a per node basis.
  • for schema changes, what do you mean ? all nodes are sharing the same schema no ? Check if the geonetwork.schema.dir variable is set in order to use the schema from the webapp and not a copy made in the datadir ?

About multinode, as discussed from time to time in 2018, we are thinking of removing this feature:

  • it does not bring much benefit (eg. since spring&JPA move, multinode mode is almost as slow as starting multiple webapps),
  • contains bugs with no project maintaining it ? I think only You and Emanuele reported issues on github about this
  • adds an extra layer of complexity (eg. no use of Spring Autowire mechanism) and when you start making changes to Java code and in particular to indexing like the move to ES.
    Also the number of known users seems quite limited and the one we know about (and initial founder) of this functionality (BRGM, PNF in France) are willing to move to GeoNetwork 3.6 version with a simplified setup and slightly different approach (work on this is planned beginning of 2019).
    So this is a topic to discuss in next developer meeting.

Cheers.

Francois

Le ven. 28 déc. 2018 à 17:47, Jo Cook <jocook@anonymised.com> a écrit :

Hi List,

I have a few questions about running geonetwork 3.4.x in multi-node mode. Is it possible to use elasticsearch and kibana in this mode- I’m worried that there will be one single index and dashboard so all administrators will see statistics for all nodes.

Secondly, when I make changes (for example to a schema) is there a way of automatically propagating those changes to all nodes, or do I have to do it manually to each one?

Many thanks

Jo

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

3

Hi Francois,

Thanks for the quick feedback. I wasn’t aware I could set a variable to ensure all nodes use the same schemas, currently I have them all getting schemas from their own data directory.

I understand about ElasticSearch- it’s a shame because it means there are two non-critical errors in the statistics page on the console, but it’s not a big deal.

Regarding deprecating multi-node- yes I’m thinking about how we can move to single nodes in future. I’ll probably want to talk to people about how they plan to do this in a cost-effective way, whether docker will be a good alternative (and if so what size server to run the containers and so on). Happy to discuss at the next developer meeting if I’m available.

Thanks again

Jo

···

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

4

Hi, yep it’s better to have schema folder set to the webapp dir if no change expected in there which is usually the case.
Same for thesaurus (with the codelist.dir variable) which could be set to a common directory not in the web app in this case as users may upload their own thesaurus.
For ES and the dashboard page, I’m not 100% sure for 3.4.x but if ES is not available it should not be displayed in the admin.

Cheers.

Francois

Le ven. 28 déc. 2018 à 18:40, Jo Cook <jocook@anonymised.com> a écrit :

Hi Francois,

Thanks for the quick feedback. I wasn’t aware I could set a variable to ensure all nodes use the same schemas, currently I have them all getting schemas from their own data directory.

I understand about ElasticSearch- it’s a shame because it means there are two non-critical errors in the statistics page on the console, but it’s not a big deal.

Regarding deprecating multi-node- yes I’m thinking about how we can move to single nodes in future. I’ll probably want to talk to people about how they plan to do this in a cost-effective way, whether docker will be a good alternative (and if so what size server to run the containers and so on). Happy to discuss at the next developer meeting if I’m available.

Thanks again

Jo

On Fri, Dec 28, 2018 at 5:24 PM Francois Prunayre <fx.prunayre@…31…> wrote:

Hi Jo and all,

About your questions:

  • current elasticsearch (ES) index config is bound to the webapplication and there is no way to define a specific ES index on a per node basis.
  • for schema changes, what do you mean ? all nodes are sharing the same schema no ? Check if the geonetwork.schema.dir variable is set in order to use the schema from the webapp and not a copy made in the datadir ?

About multinode, as discussed from time to time in 2018, we are thinking of removing this feature:

  • it does not bring much benefit (eg. since spring&JPA move, multinode mode is almost as slow as starting multiple webapps),
  • contains bugs with no project maintaining it ? I think only You and Emanuele reported issues on github about this
  • adds an extra layer of complexity (eg. no use of Spring Autowire mechanism) and when you start making changes to Java code and in particular to indexing like the move to ES.
    Also the number of known users seems quite limited and the one we know about (and initial founder) of this functionality (BRGM, PNF in France) are willing to move to GeoNetwork 3.6 version with a simplified setup and slightly different approach (work on this is planned beginning of 2019).
    So this is a topic to discuss in next developer meeting.

Cheers.

Francois

Le ven. 28 déc. 2018 à 17:47, Jo Cook <jocook@anonymised.com> a écrit :

Hi List,

I have a few questions about running geonetwork 3.4.x in multi-node mode. Is it possible to use elasticsearch and kibana in this mode- I’m worried that there will be one single index and dashboard so all administrators will see statistics for all nodes.

Secondly, when I make changes (for example to a schema) is there a way of automatically propagating those changes to all nodes, or do I have to do it manually to each one?

Many thanks

Jo

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

5

Hi Francois,

I am about to try setting the geonetwork.schema.dir variable as you suggest, but I’m not sure where I should set it. Currently I have the geonetwork.dir set in /etc/default/tomcat8, and the nodes seem to have automatically created their data directories inside there. So I have:
-Dgeonetwork.dir=/data/geonetwork in /etc/default/tomcat8

Then inside /data/geonetwork I have:
data (for the default node)
data_node1
data_node2
etc

If I set -Dgeonetwork.schema.dir in /etc/default/tomcat8 will that behave in the same way or will all nodes use the same directory? eg:
-Dgeonetwork.schema.dir=/data/geonetwork/data/config/schema_plugins

Thanks

Jo

On Fri, Dec 28, 2018 at 5:51 PM Francois Prunayre <fx.prunayre@anonymised.com> wrote:

Hi, yep it’s better to have schema folder set to the webapp dir if no change expected in there which is usually the case.
Same for thesaurus (with the codelist.dir variable) which could be set to a common directory not in the web app in this case as users may upload their own thesaurus.
For ES and the dashboard page, I’m not 100% sure for 3.4.x but if ES is not available it should not be displayed in the admin.

Cheers.

Francois

Le ven. 28 déc. 2018 à 18:40, Jo Cook <jocook@anonymised.com.> a écrit :

Hi Francois,

Thanks for the quick feedback. I wasn’t aware I could set a variable to ensure all nodes use the same schemas, currently I have them all getting schemas from their own data directory.

I understand about ElasticSearch- it’s a shame because it means there are two non-critical errors in the statistics page on the console, but it’s not a big deal.

Regarding deprecating multi-node- yes I’m thinking about how we can move to single nodes in future. I’ll probably want to talk to people about how they plan to do this in a cost-effective way, whether docker will be a good alternative (and if so what size server to run the containers and so on). Happy to discuss at the next developer meeting if I’m available.

Thanks again

Jo

On Fri, Dec 28, 2018 at 5:24 PM Francois Prunayre <fx.prunayre@anonymised.com> wrote:

Hi Jo and all,

About your questions:

  • current elasticsearch (ES) index config is bound to the webapplication and there is no way to define a specific ES index on a per node basis.
  • for schema changes, what do you mean ? all nodes are sharing the same schema no ? Check if the geonetwork.schema.dir variable is set in order to use the schema from the webapp and not a copy made in the datadir ?

About multinode, as discussed from time to time in 2018, we are thinking of removing this feature:

  • it does not bring much benefit (eg. since spring&JPA move, multinode mode is almost as slow as starting multiple webapps),
  • contains bugs with no project maintaining it ? I think only You and Emanuele reported issues on github about this
  • adds an extra layer of complexity (eg. no use of Spring Autowire mechanism) and when you start making changes to Java code and in particular to indexing like the move to ES.
    Also the number of known users seems quite limited and the one we know about (and initial founder) of this functionality (BRGM, PNF in France) are willing to move to GeoNetwork 3.6 version with a simplified setup and slightly different approach (work on this is planned beginning of 2019).
    So this is a topic to discuss in next developer meeting.

Cheers.

Francois

Le ven. 28 déc. 2018 à 17:47, Jo Cook <jocook@anonymised.com> a écrit :

Hi List,

I have a few questions about running geonetwork 3.4.x in multi-node mode. Is it possible to use elasticsearch and kibana in this mode- I’m worried that there will be one single index and dashboard so all administrators will see statistics for all nodes.

Secondly, when I make changes (for example to a schema) is there a way of automatically propagating those changes to all nodes, or do I have to do it manually to each one?

Many thanks

Jo

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

6

Hi Jo, if the schema.dir is set then all nodes should use the same location I think. After startup, you can check in admin console > information and check the path to each folders.

Francois

Le lun. 7 janv. 2019 à 15:48, Jo Cook <jocook@anonymised.com> a écrit :

Hi Francois,

I am about to try setting the geonetwork.schema.dir variable as you suggest, but I’m not sure where I should set it. Currently I have the geonetwork.dir set in /etc/default/tomcat8, and the nodes seem to have automatically created their data directories inside there. So I have:
-Dgeonetwork.dir=/data/geonetwork in /etc/default/tomcat8

Then inside /data/geonetwork I have:
data (for the default node)
data_node1
data_node2
etc

If I set -Dgeonetwork.schema.dir in /etc/default/tomcat8 will that behave in the same way or will all nodes use the same directory? eg:
-Dgeonetwork.schema.dir=/data/geonetwork/data/config/schema_plugins

Thanks

Jo

On Fri, Dec 28, 2018 at 5:51 PM Francois Prunayre <fx.prunayre@anonymised.com> wrote:

Hi, yep it’s better to have schema folder set to the webapp dir if no change expected in there which is usually the case.
Same for thesaurus (with the codelist.dir variable) which could be set to a common directory not in the web app in this case as users may upload their own thesaurus.
For ES and the dashboard page, I’m not 100% sure for 3.4.x but if ES is not available it should not be displayed in the admin.

Cheers.

Francois

Le ven. 28 déc. 2018 à 18:40, Jo Cook <jocook@anonymised.com> a écrit :

Hi Francois,

Thanks for the quick feedback. I wasn’t aware I could set a variable to ensure all nodes use the same schemas, currently I have them all getting schemas from their own data directory.

I understand about ElasticSearch- it’s a shame because it means there are two non-critical errors in the statistics page on the console, but it’s not a big deal.

Regarding deprecating multi-node- yes I’m thinking about how we can move to single nodes in future. I’ll probably want to talk to people about how they plan to do this in a cost-effective way, whether docker will be a good alternative (and if so what size server to run the containers and so on). Happy to discuss at the next developer meeting if I’m available.

Thanks again

Jo

On Fri, Dec 28, 2018 at 5:24 PM Francois Prunayre <fx.prunayre@anonymised.com> wrote:

Hi Jo and all,

About your questions:

  • current elasticsearch (ES) index config is bound to the webapplication and there is no way to define a specific ES index on a per node basis.
  • for schema changes, what do you mean ? all nodes are sharing the same schema no ? Check if the geonetwork.schema.dir variable is set in order to use the schema from the webapp and not a copy made in the datadir ?

About multinode, as discussed from time to time in 2018, we are thinking of removing this feature:

  • it does not bring much benefit (eg. since spring&JPA move, multinode mode is almost as slow as starting multiple webapps),
  • contains bugs with no project maintaining it ? I think only You and Emanuele reported issues on github about this
  • adds an extra layer of complexity (eg. no use of Spring Autowire mechanism) and when you start making changes to Java code and in particular to indexing like the move to ES.
    Also the number of known users seems quite limited and the one we know about (and initial founder) of this functionality (BRGM, PNF in France) are willing to move to GeoNetwork 3.6 version with a simplified setup and slightly different approach (work on this is planned beginning of 2019).
    So this is a topic to discuss in next developer meeting.

Cheers.

Francois

Le ven. 28 déc. 2018 à 17:47, Jo Cook <jocook@anonymised.com> a écrit :

Hi List,

I have a few questions about running geonetwork 3.4.x in multi-node mode. Is it possible to use elasticsearch and kibana in this mode- I’m worried that there will be one single index and dashboard so all administrators will see statistics for all nodes.

Secondly, when I make changes (for example to a schema) is there a way of automatically propagating those changes to all nodes, or do I have to do it manually to each one?

Many thanks

Jo

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

GeoNetwork-devel mailing list
GeoNetwork-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.

–

Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent responses at that time, please visit support.astuntechnology.com or phone our office on 01372 744009

–

Sign up to our mailing list for updates on news, products, conferences, events and training

Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18 7RL, UK
t:+44 1372 744 009 w: astuntechnology.com twitter:@astuntech

iShare - enterprise geographic intelligence platform
GeoServer, PostGIS and QGIS training
Helpdesk and customer portal

Company registration no. 5410695. Registered in England and Wales. Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no. 864201149.