[Geoserver-devel] Build speedup pull request ready

Hi,
the work on speeding up the build resulted in the following pull request, also thanks to
Christian assistance to fix some security tests:

https://github.com/geoserver/geoserver/pull/1033

On my computer the buld times of a “mvn clean install -T4 -nsu -Prelease”
went down from around 18 minutes down to around 13 minutes, not a bad speedup imho,
close to 30%

On Travis, which has to download all the geotools jars at every build,
refresh the repository cache at the beginning of each build, and so on,
we go down from 37 minutes to 30 on the openjdk builds, which is not bad either
(the last build timed 39, not sure why, the enviroment over there is really flacky,
build times go up and down as they please…).

I’ve annotated the pull request with descriptions of the changes to facilitate
life to the reviewer, feedback welcomed.

Also, I’ve found looking on the internet a suggestion to allegedly speed up the build
on windows… turns out the clean phase is pretty expensive on NTFS (blasted
file system) and you can speed it up quite a bit by simply renaming
target instead of deleting it (on Linux, no speedup achieved instead, delete
or rename is similarly fast).
One can apply this trick, and have a separate build to do the cleanups instead:
http://bosy.dailydev.org/2009/02/speed-up-your-maven-build-four-times.html

Might be worth checking out, the current build time on the windows build box
is over an hour long…

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi,
so… should I conclude everybody is happy with our current build speed? :slight_smile:

Cheers
Andrea

···

On Sun, May 3, 2015 at 9:49 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
the work on speeding up the build resulted in the following pull request, also thanks to
Christian assistance to fix some security tests:

https://github.com/geoserver/geoserver/pull/1033

On my computer the buld times of a “mvn clean install -T4 -nsu -Prelease”
went down from around 18 minutes down to around 13 minutes, not a bad speedup imho,
close to 30%

On Travis, which has to download all the geotools jars at every build,
refresh the repository cache at the beginning of each build, and so on,
we go down from 37 minutes to 30 on the openjdk builds, which is not bad either
(the last build timed 39, not sure why, the enviroment over there is really flacky,
build times go up and down as they please…).

I’ve annotated the pull request with descriptions of the changes to facilitate
life to the reviewer, feedback welcomed.

Also, I’ve found looking on the internet a suggestion to allegedly speed up the build
on windows… turns out the clean phase is pretty expensive on NTFS (blasted
file system) and you can speed it up quite a bit by simply renaming
target instead of deleting it (on Linux, no speedup achieved instead, delete
or rename is similarly fast).
One can apply this trick, and have a separate build to do the cleanups instead:
http://bosy.dailydev.org/2009/02/speed-up-your-maven-build-four-times.html

Might be worth checking out, the current build time on the windows build box
is over an hour long…

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


I will be happy to review this morning.

···

On Sun, May 3, 2015 at 9:49 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
the work on speeding up the build resulted in the following pull request, also thanks to
Christian assistance to fix some security tests:

https://github.com/geoserver/geoserver/pull/1033

On my computer the buld times of a “mvn clean install -T4 -nsu -Prelease”
went down from around 18 minutes down to around 13 minutes, not a bad speedup imho,
close to 30%

On Travis, which has to download all the geotools jars at every build,
refresh the repository cache at the beginning of each build, and so on,
we go down from 37 minutes to 30 on the openjdk builds, which is not bad either
(the last build timed 39, not sure why, the enviroment over there is really flacky,
build times go up and down as they please…).

I’ve annotated the pull request with descriptions of the changes to facilitate
life to the reviewer, feedback welcomed.

Also, I’ve found looking on the internet a suggestion to allegedly speed up the build
on windows… turns out the clean phase is pretty expensive on NTFS (blasted
file system) and you can speed it up quite a bit by simply renaming
target instead of deleting it (on Linux, no speedup achieved instead, delete
or rename is similarly fast).
One can apply this trick, and have a separate build to do the cleanups instead:
http://bosy.dailydev.org/2009/02/speed-up-your-maven-build-four-times.html

Might be worth checking out, the current build time on the windows build box
is over an hour long…

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Reviewed, you have really knocked this one out of the part - not sure I have ever seen a pull request so heavily commented.

···

On 3 May 2015 at 12:49, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
the work on speeding up the build resulted in the following pull request, also thanks to
Christian assistance to fix some security tests:

https://github.com/geoserver/geoserver/pull/1033

On my computer the buld times of a “mvn clean install -T4 -nsu -Prelease”
went down from around 18 minutes down to around 13 minutes, not a bad speedup imho,
close to 30%

On Travis, which has to download all the geotools jars at every build,
refresh the repository cache at the beginning of each build, and so on,
we go down from 37 minutes to 30 on the openjdk builds, which is not bad either
(the last build timed 39, not sure why, the enviroment over there is really flacky,
build times go up and down as they please…).

I’ve annotated the pull request with descriptions of the changes to facilitate
life to the reviewer, feedback welcomed.

Also, I’ve found looking on the internet a suggestion to allegedly speed up the build
on windows… turns out the clean phase is pretty expensive on NTFS (blasted
file system) and you can speed it up quite a bit by simply renaming
target instead of deleting it (on Linux, no speedup achieved instead, delete
or rename is similarly fast).
One can apply this trick, and have a separate build to do the cleanups instead:
http://bosy.dailydev.org/2009/02/speed-up-your-maven-build-four-times.html

Might be worth checking out, the current build time on the windows build box
is over an hour long…

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y


Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Jody Garnett

On Sun, May 10, 2015 at 4:57 AM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Reviewed, you have really knocked this one out of the part - not sure I
have ever seen a pull request so heavily commented.

As suggested in the pull request comments, I've reworked the spring bean
lazy loading so that
it's not the test that has to declare to desire eager loading, but it's the
beans that
need to be eagerly loaded that need to add lazy-init="false", like this:

https://github.com/geoserver/geoserver/pull/1033/files#diff-5b32e54953013fe49c4405ac132f9d3eR22

There is not that many needing this treatment, in general, they are beans
that
either alter their environment in the constructor (attaching listeners for
example) or
are application context aware, and doing the enviroment modification when
the
app context is set.

So, once the patch is applied, devs need to be on the lookout for this
situation, and
set the bean to not lazy init.

For the normal startup we are still going for full eager loading, it's best
to have
everything set before the OGC requests start hitting the server.
But for the tests, the speedup of lazy init, especially when running tests
from the
IDE, is too good to be left out (a single system test class run stand alone
does it job in
half of the time, 4-7 seconds instead of 10-14)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------