[Geoserver-devel] new community module workflow

GeoServer GeoServer Developer
1

Hello,

I would like to create a new community module in geoserver called "workflow".

The purpose of the module would be to create certain tasks and schedule them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

2

No objections. This seems to have some overlap with importer own list of tasks, but it’s more general.

Cheers
Andrea

···

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com.> wrote:

Hello,

I would like to create a new community module in geoserver called
“workflow”.

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:

  • update layer data from a source to local
  • synchronise data and metadata with other geoservers
  • …

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

+1 for me, sounds interesting

Ian

···

On 24 May 2017 at 13:43, Andrea Aime <andrea.aime@anonymised.com> wrote:

No objections. This seems to have some overlap with importer own list of tasks, but it’s more general.

Cheers
Andrea

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
“workflow”.

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:

  • update layer data from a source to local
  • synchronise data and metadata with other geoservers
  • …

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Ian Turton

4

Looks nice, wich framework are you going to use? There’s some stuff in the backup and restore community module working with spring-batch. Maybe you can take a look at it and see if things can be somehow homogenized.

+1 anyway

···

On May 24, 2017 14:53, “Ian Turton” <ijturton@anonymised.com> wrote:

+1 for me, sounds interesting

Ian

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On 24 May 2017 at 13:43, Andrea Aime <andrea.aime@anonymised.com> wrote:

No objections. This seems to have some overlap with importer own list of tasks, but it’s more general.

Cheers
Andrea

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Ian Turton

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
“workflow”.

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:

  • update layer data from a source to local
  • synchronise data and metadata with other geoservers
  • …

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

5

I am not against this and you already have enough +1, however it would
good to hear your thoughts on the following points:

- did you assess the overlap with existing extensions like the Importer?
- which framework do you intend to use?

There is a huge number of modules in GeoServer with overlaps and we
keep on adding new ones without harmonizing...

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
"workflow".

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

6

+1, sounds good.

Importer and the backup restore module also have some kind of tasks model. If I remember correcly the backup-restore module used a spring api that may be a useful starting point?

···

On 24 May 2017 at 04:08, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
“workflow”.

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:

  • update layer data from a source to local
  • synchronise data and metadata with other geoservers
  • …

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

7

Niels,

+1 for the concept.

Have you considered a different name? The term "workflow" is used for a variety of different purposes, including business workflows, and systems for orchestrating collections of services. For example, there are many scientific workflow management systems:
https://en.wikipedia.org/wiki/Galaxy_(computational_biology)
https://en.wikipedia.org/wiki/Apache_Taverna
https://en.wikipedia.org/wiki/Kepler_scientific_workflow_system
https://en.wikipedia.org/wiki/Scientific_workflow_system

What is the scope of your module? Does your module only schedule tasks or batches of tasks? Will it wait for task completion? Can tasks trigger other tasks? Are all the tasks local? Can tasks be invoked with parameters? Will the module manage or record task output?

A module name like "scheduled-task" might be suitable if that is the main functionality that it provides.

Also, how are tasks specified?

Kind regards,
Ben.

On 24/05/17 23:08, Niels Charlier wrote:

Hello,

I would like to create a new community module in geoserver called
"workflow".

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

8

Hello Simone, Alessio, and Jody

Thanks for your input.

I have assessed the overlap with other modules, and I don't think there is much, if any. But I am open to suggestions as always.

As I understand it, the main purpose of the Importer is publishing many layers at once. The workflow module will initially be focused on single-layer configurations. Its main purpose is to be able to manage a flow of data (and metadata) that runs over a series of servers, for example like this:

ORIGINAL DB / HARD DRIVE --> WORK-GEOSERVER --> PUBLIC GEOSERVER

and to execute this or part of this, manually or automatically if possible and desired.

In the future it could be interesting to connect the modules with each other, through the creation of importer tasks and backup/restore tasks that can be scheduled in the workflow module.

I don't think Spring batch is useful for me, because the idea is to allow users to specify tasks and batches in a GUI, and store them in DB, rather than hard-configuring the batches in the application context. Unless you think spring batch offers some useful functionality in my case also?

I will be using Quartz as scheduler framework and Hibernate for the database.

Kind Regards
Niels

On 24-05-17 15:57, Simone Giannecchini wrote:

I am not against this and you already have enough +1, however it would
good to hear your thoughts on the following points:

- did you assess the overlap with existing extensions like the Importer?
- which framework do you intend to use?

There is a huge number of modules in GeoServer with overlaps and we
keep on adding new ones without harmonizing...

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
"workflow".

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

9

Hello Ben,

Thanks for the input. See my previous email with more details. This also explains the choice of the name.

To answer your questions:

On 24-05-17 23:36, Ben Caradoc-Davies wrote:

What is the scope of your module? Does your module only schedule tasks or batches of tasks?

batches

Will it wait for task completion?

Not sure what you mean by that. Will what wait?

Can tasks trigger other tasks?

That is not something planned, but that is a theoretical possibility.

Are all the tasks local?

Not sure what you mean, they are run local but they access and write to remote resources

Can tasks be invoked with parameters?

Yes, you have task types and you specify parameters when you create a task of a certain type

Will the module manage or record task output?

Well, runs are recorded with start and end time and status. Otherwise through logging.

A module name like "scheduled-task" might be suitable if that is the main functionality that it provides.

Yes, but the main idea at this moment is that the tasks are part of workflow management. There won't be just a list of tasks. They would be grouped together in to configurations, and the main idea is that each configuration would represent a particular flow of data and metadata, i.e. a layer initially (although the general approach leaves room for other interpretations of what a configuration could be used for. Its main purpose is to provide some intuitive navigation).

Also, how are tasks specified?

Via a webGUI

Kind regards,
Ben.

On 24/05/17 23:08, Niels Charlier wrote:

Hello,

I would like to create a new community module in geoserver called
"workflow".

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

10

On 25/05/17 22:33, Niels Charlier wrote:

On 24-05-17 23:36, Ben Caradoc-Davies wrote:

Will it wait for task completion?

Not sure what you mean by that. Will what wait?

[...]

Will the module manage or record task output?

Well, runs are recorded with start and end time and status. Otherwise
through logging.

This answers my first question. :slight_smile:

A module name like "scheduled-task" might be suitable if that is the
main functionality that it provides.

Yes, but the main idea at this moment is that the tasks are part of
workflow management. There won't be just a list of tasks. They would be
grouped together in to configurations, and the main idea is that each
configuration would represent a particular flow of data and metadata,
i.e. a layer initially (although the general approach leaves room for
other interpretations of what a configuration could be used for. Its
main purpose is to provide some intuitive navigation).

It still seems to me that "workflow" is a very general term. What workflow patterns <https://en.wikipedia.org/wiki/Workflow_patterns&gt; will you support? Could future extensions support BPEL <https://en.wikipedia.org/wiki/Business_Process_Execution_Language&gt; or YAWL <https://en.wikipedia.org/wiki/YAWL&gt;?

I am not objecting to what you are trying to do, which sounds very useful, just worried that the name might mislead as to scope.

Could your module be described as "stored-procedure" or "server-procedure"?

"There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton"
https://martinfowler.com/bliki/TwoHardThings.html

Also, how are tasks specified?

Via a webGUI

What I mean to ask is whether the tasks are scripts, REST API calls, pluggable providers, or some means?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

11

On 26-05-17 03:22, Ben Caradoc-Davies wrote:

It still seems to me that "workflow" is a very general term. What workflow patterns <https://en.wikipedia.org/wiki/Workflow_patterns&gt; will you support? Could future extensions support BPEL <https://en.wikipedia.org/wiki/Business_Process_Execution_Language&gt; or YAWL <https://en.wikipedia.org/wiki/YAWL&gt;?

I am not objecting to what you are trying to do, which sounds very useful, just worried that the name might mislead as to scope.

Could your module be described as "stored-procedure" or "server-procedure"?

"There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton"
https://martinfowler.com/bliki/TwoHardThings.html

Okay Ben, you make a good case, we can pick another name to avoid confusion.
I'll have a think and discussion about it. "Stored procedure" that reminds me too much of stored procedures in databases though.

What I mean to ask is whether the tasks are scripts, REST API calls, pluggable providers, or some means?

ah, pluggable providers.

Regards
Niels

12

Hello Simone,

I understand your concern. I would also love to get rid of the various code with overlapping functionality and make the geoserver code base more elegant and robust.

In response to the approach "I'll add this one more thing because I need it." It is normal that economically speaking, additions are made to an open-source project on a need-to-have basis. The code I am writing needs to be written either way. So the only choices here are: keep the code private; create a separate (very small, unnoticed) open source project; or donate the code to the geoserver community so that other people who find it useful may use it. It was my choice to rather pick the latter.

That doesn't necessarily mean however that existing functionality should be duplicated, on the contrary. If new functionality is added, it should be optimally integrated with the old functionality.

I have thought about the overlap between importer and the new module. The only thing I see, is as you point out, the existence of a job queue. I would like to think about a way to integrate them, but there are a number of issues with that

* My job queue is far more general than importer, which only supports very specifically defined tasks. Therefore, logically, it would be importer that needs be modified to be built on top of my API rather than me building on top of importer.

* would it still make sense to have this job queue code that supports many kinds of tasks, in a module named "importer"? So then we get a new module on which importer depends rather than the other way around.

* my module will be database-driven, because my data model is far more complicated (there are tasks, batches, and configurations to organise the hundreds of tasks that may exist).
That would imply extra dependencies for hibernate. Same for the scheduler dependencies. Can I enforce these dependencies upon importer when it now works fine without?

* this can be avoided by adding extra abstraction for the job queue which may have a simple and more sophisticated implementation. Then we get two extra modules instead of one. Unless the abstraction and simple implementation could be added to one of the core modules. But then this will require a proposal etc..

So, in practice, this will mean quite a bit more work for me than if I just provide the code as a community module. But that is definitely negotiable. But I do wonder what the community thinks about the issues and options above.

Regards

Niels

On 26-05-17 16:20, Simone Giannecchini wrote:

Ciao Niels,
honestly from the description this seems to add a new module with
functionalities that should IMHO end up in the importer, possibly with
some modifications to the importer itself.

The importer is a small workflow engine and what you are trying to do
looks like an extension to the importer where, for example the ability
to run task periodically rather that on demand.
The importer already has a concept of task chain which could be extended.

As I said,I believe we are continuing to throw in new modules with
duplications and overlaps without trying to make geoserver more
robust:

- we have two mosaics with some overlaps
- we have N ways to do styling, with slight differences, why we don't
make a choice as a project we stick to that and we concentrate efforts
on implementing new feature for rendering and making the current
engine more performant/robust?
- we have 2 plugins for clustering with some overlaps
- we have N xml engines around

I would like to see some effort towards cleaning up duplications and
making geoserver a better product, but it looks like to me the
approach is rather the opposite: I'll add this one more thing because
I need it.

Btw Niels, this is not against you, it is something I have been
thinking about for a while.
According to the rules you are free to go and I won't object if you do.

That said, I think we might need to rethink the rules of comm modules
or at least to ask for a plan to be discussed a little because aside
from having a plethora of additional modules in the codebase so far I
have not seen much additional participation from outsiders (as in new
developers).

I think this is a good topic for a separate thread and a PSC meeting.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

On Thu, May 25, 2017 at 12:09 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello Simone, Alessio, and Jody

Thanks for your input.

I have assessed the overlap with other modules, and I don't think there is
much, if any. But I am open to suggestions as always.

As I understand it, the main purpose of the Importer is publishing many
layers at once. The workflow module will initially be focused on
single-layer configurations. Its main purpose is to be able to manage a flow
of data (and metadata) that runs over a series of servers, for example like
this:

ORIGINAL DB / HARD DRIVE --> WORK-GEOSERVER --> PUBLIC GEOSERVER

and to execute this or part of this, manually or automatically if possible
and desired.

In the future it could be interesting to connect the modules with each
other, through the creation of importer tasks and backup/restore tasks that
can be scheduled in the workflow module.

I don't think Spring batch is useful for me, because the idea is to allow
users to specify tasks and batches in a GUI, and store them in DB, rather
than hard-configuring the batches in the application context. Unless you
think spring batch offers some useful functionality in my case also?

I will be using Quartz as scheduler framework and Hibernate for the
database.

Kind Regards
Niels

On 24-05-17 15:57, Simone Giannecchini wrote:

I am not against this and you already have enough +1, however it would
good to hear your thoughts on the following points:

- did you assess the overlap with existing extensions like the Importer?
- which framework do you intend to use?

There is a huge number of modules in GeoServer with overlaps and we
keep on adding new ones without harmonizing...

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

On Wed, May 24, 2017 at 1:08 PM, Niels Charlier <niels@anonymised.com> wrote:

Hello,

I would like to create a new community module in geoserver called
"workflow".

The purpose of the module would be to create certain tasks and schedule
them in batches that are automatically executed.

Examples of tasks are:
- update layer data from a source to local
- synchronise data and metadata with other geoservers
- ...

and this can easily be extended upon of course.

Please let me know if this is OK.

Kind Regards

Niels

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

13

On Mon, May 29, 2017 at 2:55 PM, Niels Charlier <niels@anonymised.com> wrote:

* my module will be database-driven, because my data model is far more
complicated (there are tasks, batches, and configurations to organise the
hundreds of tasks that may exist).
That would imply extra dependencies for hibernate. Same for the scheduler
dependencies. Can I enforce these dependencies upon importer when it now
works fine without?

Hibernate... Mind cross module conflicts, monitoring and GeoFence already
use hibernate,
and possibly different versions of it

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

14

I am using the versions specified in the geoserver root pom

···

On 29-05-17 16:29, Andrea Aime wrote:

On Mon, May 29, 2017 at 2:55 PM, Niels Charlier <niels@anonymised.com> wrote:

  • my module will be database-driven, because my data model is far more complicated (there are tasks, batches, and configurations to organise the hundreds of tasks that may exist).
    That would imply extra dependencies for hibernate. Same for the scheduler dependencies. Can I enforce these dependencies upon importer when it now works fine without?

Hibernate… Mind cross module conflicts, monitoring and GeoFence already use hibernate,
and possibly different versions of it

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

15

On 30/05/17 00:55, Niels Charlier wrote:

Therefore, logically, it would be importer that needs be modified to be
built on top of my API rather than me building on top of importer.

This sounds like a great option. A modest investment in refactoring now will save us the future maintenance overhead introduced by duplication. (And I speak us someone currently fixing my own ill-considered DataUtilities duplication from five years ago.)

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

16

What is your opinion on the dependency concern?

What do you think of having the new module be called "task manager" or something like that.

Although it will contain a quite a bit of task implementation that regards remote data management specifically. Should I add two new modules though :confused:

Regards

Niels

On 30-05-17 04:05, Ben Caradoc-Davies wrote:

On 30/05/17 00:55, Niels Charlier wrote:

Therefore, logically, it would be importer that needs be modified to be
built on top of my API rather than me building on top of importer.

This sounds like a great option. A modest investment in refactoring now will save us the future maintenance overhead introduced by duplication. (And I speak us someone currently fixing my own ill-considered DataUtilities duplication from five years ago.)

Kind regards,

17

So going back to this proposal of last year. I contributed a first batch of code which is inside a new PR. Now this is a new community module that will be fully maintained by me (and fully documented soon as well) so I do not think I technically need approval for the code (although reviews/comments are always more then welcome!) but there are two small changes in the "main module" that I do need approval for. Would there someone will so kind to have a look at them (would only take a few minutes).

These are direct links to the changes in main:

https://github.com/geoserver/geoserver/pull/2759/commits/6945957a93b326ddaa39ee00c3ca4a707d663aee#diff-ed7269d9c2be90db1cbfde7d0f32d751

https://github.com/geoserver/geoserver/pull/2759/commits/6945957a93b326ddaa39ee00c3ca4a707d663aee#diff-5fac1624412cebb8475c6e0fee6fd17c

Kind Regards

Niels

On 30-05-17 16:20, Niels Charlier wrote:

What is your opinion on the dependency concern?

What do you think of having the new module be called "task manager" or something like that.

Although it will contain a quite a bit of task implementation that regards remote data management specifically. Should I add two new modules though :confused:

Regards

Niels

On 30-05-17 04:05, Ben Caradoc-Davies wrote:

On 30/05/17 00:55, Niels Charlier wrote:

Therefore, logically, it would be importer that needs be modified to be
built on top of my API rather than me building on top of importer.

This sounds like a great option. A modest investment in refactoring now will save us the future maintenance overhead introduced by duplication. (And I speak us someone currently fixing my own ill-considered DataUtilities duplication from five years ago.)

Kind regards,

------------------------------------------------------------------------------

Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel