[Geoserver-devel] Speeding up the GeoServer build by splitting some modules

GeoServer GeoServer Developer
1

Hi,
following up the geotools-devel thread on speeding up the build, I’ve made some checks on
GeoServer as well.

Compared to GeoTools the build takes longer because, among other reasons, it parallelizes a lot less.
The “chain of death” keeping the build in low parallel mode is, roughly:
main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large chunk of that time is
spent running security related tests.
I’m only semi serious here, but as far as I can see, it could be beneficial to move the securty
tests to their own module (e.g., security/core) while leaving the security classes in main.
I’m just semi-serious because I realize splitting apart the classes and their tests is not that great,
but at the same time, it could easily shave off 30-60 seconds off the critical path.

The WMS dependency onto WFS is hard to un-tangle, I’ve tried already some years ago and
eventually gave up, it would require its own mini-sprint imho… so off the board for this discussion.

Restconfig is where things get interesting and actually doable in a relatively short time.
Restconfig is a slow module to build, and has to wait onto all core OGC services to be built, but it
does not have to be that way: I propose we split the service support part into a small per service
module, e.g.:

  • gs-restconfig-wfs
  • gs-restconfig-wms
  • gs-restconfig-wcs
  • gs-restconfig-wps (missing today, no way to configure WPS from REST)
  • gs-restconfig-csw (missing today, no way to configure CSW from REST)
    Basically gs-restconfig would contain the core abstract service controller class, and the “one class modules”

would add the templates, extra code, and tests (side effect, we’d make the service rest configuration
actually pluggable).

There are also 4-5 a few tests hitting the OGC services as part of the tests, those could also be moved
to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main has completed.

The other module that should be split imho is the app-schema-test one, in the current build, and
on my current machine, it runs alone for 55 seconds after any other module has terminated building.
The reason being that it depends on too many things, wfs, wms, restconfig.
I believe the module mostly does WFS tests with a few WMS and I believe just one REST test, so can
we split it into three parts that can start running as soon as their dependencies are made available?

  • app-schema-wfs-test
  • app-schema-wms-test
  • app-schema-rest-test
    Cheers

Andrea

···

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Hi,
as with the thread on geotools-devel, there seems to be a distinct lack of enthusiams
for the topic.
Can at least get a sense if there are people opposed to the changes I’m proposing?
I’d like to know before starting to split modules apart :slight_smile:

Cheers
Andreda

···

On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
following up the geotools-devel thread on speeding up the build, I’ve made some checks on
GeoServer as well.

Compared to GeoTools the build takes longer because, among other reasons, it parallelizes a lot less.
The “chain of death” keeping the build in low parallel mode is, roughly:
main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large chunk of that time is
spent running security related tests.
I’m only semi serious here, but as far as I can see, it could be beneficial to move the securty
tests to their own module (e.g., security/core) while leaving the security classes in main.
I’m just semi-serious because I realize splitting apart the classes and their tests is not that great,
but at the same time, it could easily shave off 30-60 seconds off the critical path.

The WMS dependency onto WFS is hard to un-tangle, I’ve tried already some years ago and
eventually gave up, it would require its own mini-sprint imho… so off the board for this discussion.

Restconfig is where things get interesting and actually doable in a relatively short time.
Restconfig is a slow module to build, and has to wait onto all core OGC services to be built, but it
does not have to be that way: I propose we split the service support part into a small per service
module, e.g.:

  • gs-restconfig-wfs
  • gs-restconfig-wms
  • gs-restconfig-wcs
  • gs-restconfig-wps (missing today, no way to configure WPS from REST)
  • gs-restconfig-csw (missing today, no way to configure CSW from REST)
    Basically gs-restconfig would contain the core abstract service controller class, and the “one class modules”

would add the templates, extra code, and tests (side effect, we’d make the service rest configuration
actually pluggable).

There are also 4-5 a few tests hitting the OGC services as part of the tests, those could also be moved
to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main has completed.

The other module that should be split imho is the app-schema-test one, in the current build, and
on my current machine, it runs alone for 55 seconds after any other module has terminated building.
The reason being that it depends on too many things, wfs, wms, restconfig.
I believe the module mostly does WFS tests with a few WMS and I believe just one REST test, so can
we split it into three parts that can start running as soon as their dependencies are made available?

  • app-schema-wfs-test
  • app-schema-wms-test
  • app-schema-rest-test
    Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

I am obviously not in a place to approve / disapprove but I’ll add that your breakdown makes a lot of sense to me.

···

On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
following up the geotools-devel thread on speeding up the build, I’ve made some checks on
GeoServer as well.

Compared to GeoTools the build takes longer because, among other reasons, it parallelizes a lot less.
The “chain of death” keeping the build in low parallel mode is, roughly:
main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large chunk of that time is
spent running security related tests.
I’m only semi serious here, but as far as I can see, it could be beneficial to move the securty
tests to their own module (e.g., security/core) while leaving the security classes in main.
I’m just semi-serious because I realize splitting apart the classes and their tests is not that great,
but at the same time, it could easily shave off 30-60 seconds off the critical path.

The WMS dependency onto WFS is hard to un-tangle, I’ve tried already some years ago and
eventually gave up, it would require its own mini-sprint imho… so off the board for this discussion.

Restconfig is where things get interesting and actually doable in a relatively short time.
Restconfig is a slow module to build, and has to wait onto all core OGC services to be built, but it
does not have to be that way: I propose we split the service support part into a small per service
module, e.g.:

  • gs-restconfig-wfs
  • gs-restconfig-wms
  • gs-restconfig-wcs
  • gs-restconfig-wps (missing today, no way to configure WPS from REST)
  • gs-restconfig-csw (missing today, no way to configure CSW from REST)
    Basically gs-restconfig would contain the core abstract service controller class, and the “one class modules”

would add the templates, extra code, and tests (side effect, we’d make the service rest configuration
actually pluggable).

There are also 4-5 a few tests hitting the OGC services as part of the tests, those could also be moved
to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main has completed.

The other module that should be split imho is the app-schema-test one, in the current build, and
on my current machine, it runs alone for 55 seconds after any other module has terminated building.
The reason being that it depends on too many things, wfs, wms, restconfig.
I believe the module mostly does WFS tests with a few WMS and I believe just one REST test, so can
we split it into three parts that can start running as soon as their dependencies are made available?

  • app-schema-wfs-test
  • app-schema-wms-test
  • app-schema-rest-test
    Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

I certainly have no objections to things being faster.

My only concern was on the geotools side, if there was an API change. I don’t see that as a consideration on geoserver though (application vs library).

I’m not really able to help with it at the moment though, although I may be able to apply some test effort.

Brad

From: Andrea Aime [mailto:andrea.aime@anonymised.com]
Sent: Saturday, 14 October 2017 2:30 AM
To: Geoserver-devel geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Speeding up the GeoServer build by splitting some modules

Hi,

as with the thread on geotools-devel, there seems to be a distinct lack of enthusiams

for the topic.

Can at least get a sense if there are people opposed to the changes I’m proposing?

I’d like to know before starting to split modules apart :slight_smile:

Cheers

Andreda

On Wed, Oct 11, 2017 at 6:36 PM, Andrea Aime <andrea.aime@anonymised.com.> wrote:

Hi,

following up the geotools-devel thread on speeding up the build, I’ve made some checks on

GeoServer as well.

Compared to GeoTools the build takes longer because, among other reasons, it parallelizes a lot less.

The “chain of death” keeping the build in low parallel mode is, roughly:

main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large chunk of that time is

spent running security related tests.

I’m only semi serious here, but as far as I can see, it could be beneficial to move the securty

tests to their own module (e.g., security/core) while leaving the security classes in main.

I’m just semi-serious because I realize splitting apart the classes and their tests is not that great,

but at the same time, it could easily shave off 30-60 seconds off the critical path.

The WMS dependency onto WFS is hard to un-tangle, I’ve tried already some years ago and

eventually gave up, it would require its own mini-sprint imho… so off the board for this discussion.

Restconfig is where things get interesting and actually doable in a relatively short time.

Restconfig is a slow module to build, and has to wait onto all core OGC services to be built, but it

does not have to be that way: I propose we split the service support part into a small per service

module, e.g.:

  • gs-restconfig-wfs
  • gs-restconfig-wms
  • gs-restconfig-wcs
  • gs-restconfig-wps (missing today, no way to configure WPS from REST)
  • gs-restconfig-csw (missing today, no way to configure CSW from REST)

Basically gs-restconfig would contain the core abstract service controller class, and the “one class modules”

would add the templates, extra code, and tests (side effect, we’d make the service rest configuration

actually pluggable).

There are also 4-5 a few tests hitting the OGC services as part of the tests, those could also be moved

to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main has completed.

The other module that should be split imho is the app-schema-test one, in the current build, and

on my current machine, it runs alone for 55 seconds after any other module has terminated building.

The reason being that it depends on too many things, wfs, wms, restconfig.

I believe the module mostly does WFS tests with a few WMS and I believe just one REST test, so can

we split it into three parts that can start running as soon as their dependencies are made available?

  • app-schema-wfs-test
  • app-schema-wms-test
  • app-schema-rest-test

Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

5

Thinking about splitting up rest-config …

We ship the rest api out of the box currently I have not heard of any deployments where people have used our modular system make a distribution that does not include the rest api?

The thing I like most about splitting up rest-config is it would allow the application to be more modular. If the rest-api is always present is it worth adding the “rest-config” into the individual gs-wfs and gs-wms modules?

···

On 11 October 2017 at 09:36, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
following up the geotools-devel thread on speeding up the build, I’ve made some checks on
GeoServer as well.

Compared to GeoTools the build takes longer because, among other reasons, it parallelizes a lot less.
The “chain of death” keeping the build in low parallel mode is, roughly:
main->wfs->wms->restconfig->app-schema-test

The main module takes here 1:20, and everything else depends on it. A large chunk of that time is
spent running security related tests.
I’m only semi serious here, but as far as I can see, it could be beneficial to move the securty
tests to their own module (e.g., security/core) while leaving the security classes in main.
I’m just semi-serious because I realize splitting apart the classes and their tests is not that great,
but at the same time, it could easily shave off 30-60 seconds off the critical path.

The WMS dependency onto WFS is hard to un-tangle, I’ve tried already some years ago and
eventually gave up, it would require its own mini-sprint imho… so off the board for this discussion.

Restconfig is where things get interesting and actually doable in a relatively short time.
Restconfig is a slow module to build, and has to wait onto all core OGC services to be built, but it
does not have to be that way: I propose we split the service support part into a small per service
module, e.g.:

  • gs-restconfig-wfs
  • gs-restconfig-wms
  • gs-restconfig-wcs
  • gs-restconfig-wps (missing today, no way to configure WPS from REST)
  • gs-restconfig-csw (missing today, no way to configure CSW from REST)
    Basically gs-restconfig would contain the core abstract service controller class, and the “one class modules”

would add the templates, extra code, and tests (side effect, we’d make the service rest configuration
actually pluggable).

There are also 4-5 a few tests hitting the OGC services as part of the tests, those could also be moved
to the service specific restconfig module.

This way the main gs-restconfig module could start building as soon as gs-main has completed.

The other module that should be split imho is the app-schema-test one, in the current build, and
on my current machine, it runs alone for 55 seconds after any other module has terminated building.
The reason being that it depends on too many things, wfs, wms, restconfig.
I believe the module mostly does WFS tests with a few WMS and I believe just one REST test, so can
we split it into three parts that can start running as soon as their dependencies are made available?

  • app-schema-wfs-test
  • app-schema-wms-test
  • app-schema-rest-test
    Cheers

Andrea

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

6

I do that regularly for customers, but those are custom builds for high security environments: no ui, no rest, no way to change the fixed configuration.

In other cases I have setups that a more micro-service oriented where the war supposedly contains a single service.

Finally, the UI is already split on a per service modules, why not have REST align?

Besides the above feedback, even if rest config was always in, the per service modules would be in the core packaging anyways of course. So, where would be the harm, and how would it offset the benefit of a faster build to the point that we should abandon the idea? (your feedback sounds like a -1)

Regards
Andrea

···

Il 13 ott 2017 10:13 PM, “Jody Garnett” <jody.garnett@anonymised.com> ha scritto:

Thinking about splitting up rest-config …

We ship the rest api out of the box currently I have not heard of any deployments where people have used our modular system make a distribution that does not include the rest api?

The thing I like most about splitting up rest-config is it would allow the application to be more modular. If the rest-api is always present is it worth adding the “rest-config” into the individual gs-wfs and gs-wms modules?

7

My feedback was not a -1, given that you have a solid benefit from the modular architecture (secure env and micro deployment) I am firmly behind breaking up rest-config into modules :slight_smile:

···

On 13 October 2017 at 13:35, Andrea Aime <andrea.aime@anonymised.com> wrote:

I do that regularly for customers, but those are custom builds for high security environments: no ui, no rest, no way to change the fixed configuration.

In other cases I have setups that a more micro-service oriented where the war supposedly contains a single service.

Finally, the UI is already split on a per service modules, why not have REST align?

Besides the above feedback, even if rest config was always in, the per service modules would be in the core packaging anyways of course. So, where would be the harm, and how would it offset the benefit of a faster build to the point that we should abandon the idea? (your feedback sounds like a -1)

Regards

Andrea

–
Jody Garnett

Il 13 ott 2017 10:13 PM, “Jody Garnett” <jody.garnett@anonymised.com> ha scritto:

Thinking about splitting up rest-config …

We ship the rest api out of the box currently I have not heard of any deployments where people have used our modular system make a distribution that does not include the rest api?

The thing I like most about splitting up rest-config is it would allow the application to be more modular. If the rest-api is always present is it worth adding the “rest-config” into the individual gs-wfs and gs-wms modules?

8

And here we go:

https://github.com/geoserver/geoserver/pull/2599

With those changes I can build GeoServer locally within 6 minutes.
The two big speedups have been:

  • Splitting restconfig service into their own separate modules
  • Moving the gs-main security tests into their own separate module (this one alone cuts 50 seconds away from the build of both gs-main and the overall build time)
    In the end I did not touch the app-schema tests as with the new setup they are finishing just a few seconds after gs-web-gwc. Currently the “slow” part is wfs->wms->gwc, to make it faster one would need to untangle wms from its wfs dependencies (which is not trivial).

I’ve noticed that importer depends for no particular good reason on gs-wms (just to run a test against wms getCapabilities, but the same could be done listing directly from the catalog), there might be some minor build times improvement in there, but right now I have no time… I might have a quick look next week :slight_smile:

Cheers
Andrea

···

On Sat, Oct 14, 2017 at 1:49 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

My feedback was not a -1, given that you have a solid benefit from the modular architecture (secure env and micro deployment) I am firmly behind breaking up rest-config into modules :slight_smile:

–

–
Jody Garnett

On 13 October 2017 at 13:35, Andrea Aime <andrea.aime@anonymised.com> wrote:

I do that regularly for customers, but those are custom builds for high security environments: no ui, no rest, no way to change the fixed configuration.

In other cases I have setups that a more micro-service oriented where the war supposedly contains a single service.

Finally, the UI is already split on a per service modules, why not have REST align?

Besides the above feedback, even if rest config was always in, the per service modules would be in the core packaging anyways of course. So, where would be the harm, and how would it offset the benefit of a faster build to the point that we should abandon the idea? (your feedback sounds like a -1)

Regards

Andrea

Il 13 ott 2017 10:13 PM, “Jody Garnett” <jody.garnett@anonymised.com> ha scritto:

Thinking about splitting up rest-config …

We ship the rest api out of the box currently I have not heard of any deployments where people have used our modular system make a distribution that does not include the rest api?

The thing I like most about splitting up rest-config is it would allow the application to be more modular. If the rest-api is always present is it worth adding the “rest-config” into the individual gs-wfs and gs-wms modules?

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.