[Geoserver-devel] Thinking out loud... splitting modules to get faster build times?

Andrea_Aime4 · November 1, 2018, 6:48pm

Hi,
I was looking into build times, and trying to see if there is anything doable to reduce them.
On a serial build I’ve found nothing, tried to tune the JVM heap, the garbage collector, run test with a profiler, nothing helped.

However, on a parallel build, there are “flat CPU” parts of the build that enjoy little parallelization.
On that shows up evidently is, towards the end, app-schema tests, they take a lot of time, and end up being at the
end because they depend both on wms and wfs, which are the main drivers of our build (the slow part
is main->wfs->wms, after main there is parallelilzation, then we have to wait for wfs to finish, parallelization
again, then wms to finish, final parallelization).

The app-schema ones have a small number of wms tests, but are waiting for wms to build because of those.
So I’m asking to the app-schema experts, would it be hard to have a app-schema-wfs-test and a app-schema-wms-test,
making the bulk of app-schema test run as soon as wfs is ready?
I see there are also dependencies on web-ui and restconfig, those could maybe also be split in their own little modules

The other “beast” that could enjoy some splitting is gs-main, the security portion could maybe be separated out, but
not sure there are many modules that could run without depending also on it (which would defeat the purpose of
splitting main). Has anyone ever looked into it?

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Gabriel_Roldan2 · November 1, 2018, 10:08pm

Hello,

I can’t comment on app schema tests nor splitting security out of main, although the latest sounds like a good idea and will probably lead to a bit of decoupling which should be a good thing, say you want a stripped down version of geoserver with no security at all, then you could go for it. But maybe I’m just talking out of ignorance here.

What I do want to bring to the table is two things:

On build.geoserver.org, there’s a tmpfs mount at /mnt/memtmp that can be used to speed up the build by means of avoiding tests to hit the disk so much. We’ve done that for the geogig builds and it helped a lot. Would just need to add the following JVM argument to the build: -Djava.io.tmpdir=/mnt/memtmp/geoserver
Last time I checked (a rather long time ago) the build left quite a bit of garbage in /tmp (though not sure of it was geoserver’s or geotools’ build that left the most). This could also be an opportunity to avoid flooding the build server with that many remanents (it ran out of disk space once upon a time due to that). To do so we could run a post build shell script that just removed everything in /mnt/memtmp/geoserver, if no two builds will ever run in parallel. Otherwise it’d be cool if the tmp directory could be parametrized on a per build basis, like in running a pre build script that creates /mnt/memtmp/${build.number} or so, and a post script that removes it.
The other thing I’d like to mention, although a bit off topic, is a what if the main codebase is reduced. That is, IMHO, the current policy for both gt and gs of allowing almost anyone to create almost any community module on the main codebase is rather archaic. What do you guys think of creating sibling projects under each project github organization to hold those (that is, github.com/geoserver/community and github.com/geotools/unsupported)? After all, one should (would, could) keep an unofficial project/module as an independent project until deemed good enough to ask to be included as an unsupported module, and then just once it’s good enough for the main codebase be moved to it, instead of flooding it with runaway unsupported modules?
I don’t know, I think that could work out pretty well.

Just my 2c.

Cheers,
Gabriel

···

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

Nuno_Oliveira1 · November 2, 2018, 10:54am

Hi Andrea,

The app-schema ones have a small number of wms tests, but are waiting for wms to build because of those.
So I’m asking to the app-schema experts, would it be hard to have a app-schema-wfs-test and a app-schema-wms-test,
making the bulk of app-schema test run as soon as wfs is ready?

It could be done, I will try to have a look this weekend.
Do you have any idea of how much will be the build time gain here ?

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Andrea_Aime4 · November 3, 2018, 11:14am

On Fri, Nov 2, 2018 at 11:54 AM Nuno Oliveira <nuno.oliveira@anonymised.com> wrote:

It could be done, I will try to have a look this weekend.

Thanks!

Do you have any idea of how much will be the build time gain here ?

I’m not sure yet, but here is what I have (spoiler, towards the end I show it might not actually be worth it)

To build I normally use the takari smart builder ( https://github.com/takari/takari-smart-builder ) which
besides allowing better build parallelization and critical path scheduling, logs some interesting info about the current build queue.
I’ve grepped it out of a build, this is the result:

mvn clean install -T8 -fae -nsu --builder smart -Prelease 2>/dev/null | grep “Builder state”

7316 [INFO] Builder state: blocked=97 finished=1 ready-or-running=2 [org.geoserver:web org.geoserver:gs-platform]

7957 [INFO] Builder state: blocked=97 finished=2 ready-or-running=1 [org.geoserver:gs-platform]
18795 [INFO] Builder state: blocked=94 finished=3 ready-or-running=3 [org.geoserver:community org.geoserver:gs-ows org.geoserver:extension]
19266 [INFO] Builder state: blocked=80 finished=4 ready-or-running=16
19302 [INFO] Builder state: blocked=80 finished=5 ready-or-running=15
20072 [INFO] Builder state: blocked=80 finished=6 ready-or-running=14
20076 [INFO] Builder state: blocked=80 finished=7 ready-or-running=13
20076 [INFO] Builder state: blocked=80 finished=8 ready-or-running=12
20098 [INFO] Builder state: blocked=80 finished=9 ready-or-running=11
20156 [INFO] Builder state: blocked=80 finished=10 ready-or-running=10
20291 [INFO] Builder state: blocked=80 finished=11 ready-or-running=9
20756 [INFO] Builder state: blocked=80 finished=12 ready-or-running=8
20775 [INFO] Builder state: blocked=80 finished=13 ready-or-running=7 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-imagemosaic-jdbc org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-db2 org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20777 [INFO] Builder state: blocked=80 finished=14 ready-or-running=6 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-imagemosaic-jdbc org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20801 [INFO] Builder state: blocked=80 finished=15 ready-or-running=5 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20807 [INFO] Builder state: blocked=80 finished=16 ready-or-running=4 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20896 [INFO] Builder state: blocked=80 finished=17 ready-or-running=3 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver:gs-ows]
21356 [INFO] Builder state: blocked=80 finished=18 ready-or-running=2 [org.geoserver.extension:gs-gdal org.geoserver:gs-ows]
22511 [INFO] Builder state: blocked=80 finished=19 ready-or-running=1 [org.geoserver:gs-ows]
25191 [INFO] Builder state: blocked=79 finished=20 ready-or-running=1 [org.geoserver:gs-main]
77359 [INFO] Builder state: blocked=70 finished=21 ready-or-running=9
78200 [INFO] Builder state: blocked=69 finished=22 ready-or-running=9
78310 [INFO] Builder state: blocked=67 finished=23 ready-or-running=10
82618 [INFO] Builder state: blocked=67 finished=24 ready-or-running=9
84323 [INFO] Builder state: blocked=67 finished=25 ready-or-running=8
85527 [INFO] Builder state: blocked=65 finished=26 ready-or-running=9
104943 [INFO] Builder state: blocked=65 finished=27 ready-or-running=8
105807 [INFO] Builder state: blocked=65 finished=28 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver.csw:gs-csw-api org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-rest org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
106272 [INFO] Builder state: blocked=64 finished=29 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver.csw:gs-csw-simple-store org.geoserver:gs-rest org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
111240 [INFO] Builder state: blocked=63 finished=30 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.csw:gs-csw-simple-store org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
126473 [INFO] Builder state: blocked=62 finished=31 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core org.geoserver:gs-wcs1_1]
139478 [INFO] Builder state: blocked=61 finished=32 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core]
142992 [INFO] Builder state: blocked=61 finished=33 ready-or-running=6 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core]
164516 [INFO] Builder state: blocked=61 finished=34 ready-or-running=5 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
183365 [INFO] Builder state: blocked=55 finished=35 ready-or-running=10
184296 [INFO] Builder state: blocked=55 finished=36 ready-or-running=9
200564 [INFO] Builder state: blocked=55 finished=37 ready-or-running=8
200568 [INFO] Builder state: blocked=55 finished=38 ready-or-running=7 [org.geoserver.extension:gs-grib org.geoserver.extension:gs-teradata org.geoserver.extension:gs-printing org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
201993 [INFO] Builder state: blocked=55 finished=39 ready-or-running=6 [org.geoserver.extension:gs-grib org.geoserver.extension:gs-teradata org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
202183 [INFO] Builder state: blocked=55 finished=40 ready-or-running=5 [org.geoserver.extension:gs-grib org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
203917 [INFO] Builder state: blocked=55 finished=41 ready-or-running=4 [org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
205539 [INFO] Builder state: blocked=54 finished=42 ready-or-running=4 [org.geoserver:gs-wfs org.geoserver.extension:gs-wcs2_0-eo-core org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
207482 [INFO] Builder state: blocked=45 finished=43 ready-or-running=12
210406 [INFO] Builder state: blocked=43 finished=44 ready-or-running=13
211602 [INFO] Builder state: blocked=38 finished=45 ready-or-running=17
232568 [INFO] Builder state: blocked=38 finished=46 ready-or-running=16
233239 [INFO] Builder state: blocked=37 finished=47 ready-or-running=16
233768 [INFO] Builder state: blocked=37 finished=48 ready-or-running=15
235338 [INFO] Builder state: blocked=37 finished=49 ready-or-running=14
245546 [INFO] Builder state: blocked=37 finished=50 ready-or-running=13
246113 [INFO] Builder state: blocked=37 finished=51 ready-or-running=12
252089 [INFO] Builder state: blocked=37 finished=52 ready-or-running=11
253813 [INFO] Builder state: blocked=37 finished=53 ready-or-running=10
255022 [INFO] Builder state: blocked=37 finished=54 ready-or-running=9
257966 [INFO] Builder state: blocked=37 finished=55 ready-or-running=8
258552 [INFO] Builder state: blocked=37 finished=56 ready-or-running=7 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.web:gs-web-rest org.geoserver.extension:gs-web-wcs2_0-eo org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
262807 [INFO] Builder state: blocked=37 finished=57 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.extension:gs-web-wcs2_0-eo org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
266017 [INFO] Builder state: blocked=37 finished=58 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
266249 [INFO] Builder state: blocked=37 finished=59 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
267150 [INFO] Builder state: blocked=37 finished=60 ready-or-running=3 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-wps-core]
282823 [INFO] Builder state: blocked=34 finished=61 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-dxf-wps org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.extension:gs-ogr-wps]
290936 [INFO] Builder state: blocked=34 finished=62 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.extension:gs-ogr-wps]
292186 [INFO] Builder state: blocked=31 finished=63 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.extension:gs-security org.geoserver.web:gs-web-sec-jdbc org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
292651 [INFO] Builder state: blocked=29 finished=64 ready-or-running=7 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-web-sec org.geoserver.extension:gs-ogr-wps]
293199 [INFO] Builder state: blocked=29 finished=65 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
293995 [INFO] Builder state: blocked=29 finished=66 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
295786 [INFO] Builder state: blocked=29 finished=67 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.web:gs-web-sec-ldap]
298512 [INFO] Builder state: blocked=28 finished=68 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-web-sec-cas org.geoserver.web:gs-web-sec-ldap]
300423 [INFO] Builder state: blocked=28 finished=69 ready-or-running=3 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.web:gs-web-sec-ldap]
302346 [INFO] Builder state: blocked=28 finished=70 ready-or-running=2 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc]
319821 [INFO] Builder state: blocked=16 finished=71 ready-or-running=13
320386 [INFO] Builder state: blocked=15 finished=72 ready-or-running=13
348110 [INFO] Builder state: blocked=15 finished=73 ready-or-running=12
353240 [INFO] Builder state: blocked=12 finished=74 ready-or-running=14
356878 [INFO] Builder state: blocked=11 finished=75 ready-or-running=14
357299 [INFO] Builder state: blocked=11 finished=76 ready-or-running=13
364581 [INFO] Builder state: blocked=10 finished=77 ready-or-running=13
371091 [INFO] Builder state: blocked=10 finished=78 ready-or-running=12
380418 [INFO] Builder state: blocked=9 finished=79 ready-or-running=12
389545 [INFO] Builder state: blocked=9 finished=80 ready-or-running=11
395658 [INFO] Builder state: blocked=9 finished=81 ready-or-running=10
396566 [INFO] Builder state: blocked=9 finished=82 ready-or-running=9
406503 [INFO] Builder state: blocked=6 finished=83 ready-or-running=11
409425 [INFO] Builder state: blocked=6 finished=84 ready-or-running=10
409602 [INFO] Builder state: blocked=6 finished=85 ready-or-running=9
410405 [INFO] Builder state: blocked=6 finished=86 ready-or-running=8
417603 [INFO] Builder state: blocked=6 finished=87 ready-or-running=7 [org.geoserver.importer:gs-importer-web org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-imagemap]
421548 [INFO] Builder state: blocked=6 finished=88 ready-or-running=6 [org.geoserver.importer:gs-importer-web org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
426601 [INFO] Builder state: blocked=6 finished=89 ready-or-running=5 [org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
426636 [INFO] Builder state: blocked=6 finished=90 ready-or-running=4 [org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
430242 [INFO] Builder state: blocked=6 finished=91 ready-or-running=3 [org.geoserver.importer:gs-importer-rest org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
443521 [INFO] Builder state: blocked=6 finished=92 ready-or-running=2 [org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
456022 [INFO] Builder state: blocked=2 finished=93 ready-or-running=5 [org.geoserver.extension:gs-ysld org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-inspire]
470715 [INFO] Builder state: blocked=2 finished=94 ready-or-running=4 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-inspire]
475546 [INFO] Builder state: blocked=2 finished=95 ready-or-running=3 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-inspire]
478682 [INFO] Builder state: blocked=2 finished=96 ready-or-running=2 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-inspire]
484459 [INFO] Builder state: blocked=1 finished=97 ready-or-running=2 [org.geoserver.web:gs-web-gwc org.geoserver:gs-release]
485380 [INFO] Builder state: blocked=1 finished=98 ready-or-running=1 [org.geoserver.web:gs-web-gwc]
491967 [INFO] Builder state: blocked=0 finished=99 ready-or-running=1
496308 [INFO] Builder state: blocked=0 finished=100 ready-or-running=0

The output starts with a time in milliseconds from the begin of the build, and then info about how many are runnable or running (and I was using intellij on the side, so times are a little longer than usual).
A new message is generated when a module end building.
Unfortunately the output gives info only when there are less runnables than one would like to, but generally speaking, I see the
my CPU (ryzen 1700x, 8 cores) “suffer” (being under-utilized) when the number of running processes goes below 5 or 6.

I’ve highlighted a few sections that show a criticality:

When gs-main is reached, there is a 50 seconds “stop the world” waiting for it to finish (there are a couple of security tests in there that are dead slow)
The gs-wfs starts building (cannot be seen, but it’s there) and the other parallel jobs finish before it finishes, bringing down the concurrent jobs to 4, and then after it lots of parallelization again
After gs-wfs the other beast is gs-wms, we have to wait for it to finish building before another large parallelization happens
App-schema starts building in that phase where there are lots of modules active and finishes almost at the end (parallel builds are not fully reproducable, I often see it finishing last) … my hope is that by splitting it it would start running before and leave more breathing room towards the end for the CPUs to go higher freq
I’m focusing on app-schema tests because it seems the easiest to take apart and in common builds I often see it running towards the end, but major benefits would be reaped by having gs-main build faster, and also having gs-wms not depend on gs-wfs (GetFeatureInfo does GML encoding unfortunately, not sure there are other dependencies… maybe we could have a gs-wms-info module that depends on a simpler gs-wms and gs-wfs… something to try out) so that gs-gwc can start its build sooner. App-schema-test would still depend on the gs-wms-info one, since it’s doing GetFeatureInfo on complex features.

gs-gwc is another module with very slow tests (the is at least one repeating the data dir setup, instead of sharing it) and in a critical position, that could use some work.

After writing this I thought to also check a -T4 build, since most machines are still stuck on 4 cores (including our build servers).
Long story short, a -T4 build is almost fully loaded all the time, and the above changes to app-schema would likely do little to speed it up, that build would likely
be made faster only by focusing on slow tests and making them faster.
I’ll have a look at chasing them down in a sequential build, and follow up.

Cheers
Andrea

–

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Andrea_Aime4 · November 3, 2018, 11:49am

Hello,

I can’t comment on app schema tests nor splitting security out of main, although the latest sounds like a good idea and will probably lead to a bit of decoupling which should be a good thing, say you want a stripped down version of geoserver with no security at all, then you could go for it. But maybe I’m just talking out of ignorance here.

Eh, I agree it would be a good idea, but like you I’m not familiar enough with it to tell if that would be feasible, or not.

What I do want to bring to the table is two things:

On build.geoserver.org, there’s a tmpfs mount at /mnt/memtmp that can be used to speed up the build by means of avoiding tests to hit the disk so much. We’ve done that for the geogig builds and it helped a lot. Would just need to add the following JVM argument to the build: -Djava.io.tmpdir=/mnt/memtmp/geoserver
Last time I checked (a rather long time ago) the build left quite a bit of garbage in /tmp (though not sure of it was geoserver’s or geotools’ build that left the most).

Believe it’s mostly GeoTools, but might be wrong. The quantity of garbage left in /tmp has been reduced a lot compared to many years ago, but there is still a good bunch lying there
that one day or the other we should try to kill.

This could also be an opportunity to avoid flooding the build server with that many remanents (it ran out of disk space once upon a time due to that). To do so we could run a post build shell script that just removed everything in /mnt/memtmp/geoserver, if no two builds will ever run in parallel. Otherwise it’d be cool if the tmp directory could be parametrized on a per build basis, like in running a pre build script that creates /mnt/memtmp/${build.number} or so, and a post script that removes it.

Right. The GeoServer build creates and destroys a lot of data dirs, one per test class normally, that all live in ./target/ (current module’s).
Profiling surefire on a full module build (tried both wms and wfs) did not show IO to be dominant though, the results are very spread out
and the only consistent bits showing in every module are related to classloading, the rest tend to change module by module.
The first I/O related bit is like in 10-15th position (but I have a fast SSD in this machine, the situation might be different on others).

However, since most temp directory handling is centralized inside SystemTestData, I’ve created a 1GB ramdisk and
had that class create all data dirs in it, then made SystemTestData create the data dirs in it.
Doing a -T6 builds (seems to be the fastest on my machine, at around 440seconds unmodified), I got a 430s build time…
an improvement of 10 seconds can be random swinging, but even if it was not, does not look like worthwhile

The other thing I’d like to mention, although a bit off topic, is a what if the main codebase is reduced. That is, IMHO, the current policy for both gt and gs of allowing almost anyone to create almost any community module on the main codebase is rather archaic. What do you guys think of creating sibling projects under each project github organization to hold those (that is, github.com/geoserver/community and github.com/geotools/unsupported)? After all, one should (would, could) keep an unofficial project/module as an independent project until deemed good enough to ask to be included as an unsupported module, and then just once it’s good enough for the main codebase be moved to it, instead of flooding it with runaway unsupported modules?

I don’t know, I think that could work out pretty well.

Community modules are not built on a normal build on a dev machine, the customary command line is “mvn clean install -TX -nsu -Prelease” which leaves them out.
On the build server they do account for 8 minutes out of 31 total on the jenkins build server, and only 6 out of 41 minutes on Travis,
however at the moment I’m mostly concerned about developer productivity, as everybody participating in the JDK 11 code sprint has experienced,
having to wait 7-9 minutes for a full GeoServer build is a tad too much imho (timings out of my machine, overall build time varies a lot
depending on what else I’m doing during the build)

Cheers
Andrea

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Nuno_Oliveira1 · November 3, 2018, 2:45pm

Thanks for the feedback, start having a look at this

On Sat, 2018-11-03 at 12:14 +0100, Andrea Aime wrote:

On Fri, Nov 2, 2018 at 11:54 AM Nuno Oliveira <nuno.oliveira@anonymised.com> wrote:

It could be done, I will try to have a look this weekend.

Thanks!

Do you have any idea of how much will be the build time gain here ?

I’m not sure yet, but here is what I have (spoiler, towards the end I show it might not actually be worth it)

To build I normally use the takari smart builder ( https://github.com/takari/takari-smart-builder ) which
besides allowing better build parallelization and critical path scheduling, logs some interesting info about the current build queue.
I’ve grepped it out of a build, this is the result:

mvn clean install -T8 -fae -nsu --builder smart -Prelease 2>/dev/null | grep “Builder state”

7316 [INFO] Builder state: blocked=97 finished=1 ready-or-running=2 [org.geoserver:web org.geoserver:gs-platform]

7957 [INFO] Builder state: blocked=97 finished=2 ready-or-running=1 [org.geoserver:gs-platform]
18795 [INFO] Builder state: blocked=94 finished=3 ready-or-running=3 [org.geoserver:community org.geoserver:gs-ows org.geoserver:extension]
19266 [INFO] Builder state: blocked=80 finished=4 ready-or-running=16
19302 [INFO] Builder state: blocked=80 finished=5 ready-or-running=15
20072 [INFO] Builder state: blocked=80 finished=6 ready-or-running=14
20076 [INFO] Builder state: blocked=80 finished=7 ready-or-running=13
20076 [INFO] Builder state: blocked=80 finished=8 ready-or-running=12
20098 [INFO] Builder state: blocked=80 finished=9 ready-or-running=11
20156 [INFO] Builder state: blocked=80 finished=10 ready-or-running=10
20291 [INFO] Builder state: blocked=80 finished=11 ready-or-running=9
20756 [INFO] Builder state: blocked=80 finished=12 ready-or-running=8
20775 [INFO] Builder state: blocked=80 finished=13 ready-or-running=7 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-imagemosaic-jdbc org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-db2 org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20777 [INFO] Builder state: blocked=80 finished=14 ready-or-running=6 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-imagemosaic-jdbc org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20801 [INFO] Builder state: blocked=80 finished=15 ready-or-running=5 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver.extension:gs-sqlserver org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20807 [INFO] Builder state: blocked=80 finished=16 ready-or-running=4 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver.extension:gs-jp2k org.geoserver:gs-ows]
20896 [INFO] Builder state: blocked=80 finished=17 ready-or-running=3 [org.geoserver.extension:gs-gdal org.geoserver.extension:gs-mysql org.geoserver:gs-ows]
21356 [INFO] Builder state: blocked=80 finished=18 ready-or-running=2 [org.geoserver.extension:gs-gdal org.geoserver:gs-ows]
22511 [INFO] Builder state: blocked=80 finished=19 ready-or-running=1 [org.geoserver:gs-ows]
25191 [INFO] Builder state: blocked=79 finished=20 ready-or-running=1 [org.geoserver:gs-main]
77359 [INFO] Builder state: blocked=70 finished=21 ready-or-running=9
78200 [INFO] Builder state: blocked=69 finished=22 ready-or-running=9
78310 [INFO] Builder state: blocked=67 finished=23 ready-or-running=10
82618 [INFO] Builder state: blocked=67 finished=24 ready-or-running=9
84323 [INFO] Builder state: blocked=67 finished=25 ready-or-running=8
85527 [INFO] Builder state: blocked=65 finished=26 ready-or-running=9
104943 [INFO] Builder state: blocked=65 finished=27 ready-or-running=8
105807 [INFO] Builder state: blocked=65 finished=28 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver.csw:gs-csw-api org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-rest org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
106272 [INFO] Builder state: blocked=64 finished=29 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver.csw:gs-csw-simple-store org.geoserver:gs-rest org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
111240 [INFO] Builder state: blocked=63 finished=30 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.csw:gs-csw-simple-store org.geoserver.security:gs-security-tests org.geoserver:gs-wcs1_1]
126473 [INFO] Builder state: blocked=62 finished=31 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core org.geoserver:gs-wcs1_1]
139478 [INFO] Builder state: blocked=61 finished=32 ready-or-running=7 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-wcs1_0 org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core]
142992 [INFO] Builder state: blocked=61 finished=33 ready-or-running=6 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests org.geoserver.csw:gs-csw-core]
164516 [INFO] Builder state: blocked=61 finished=34 ready-or-running=5 [org.geoserver.web:gs-web-core org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
183365 [INFO] Builder state: blocked=55 finished=35 ready-or-running=10
184296 [INFO] Builder state: blocked=55 finished=36 ready-or-running=9
200564 [INFO] Builder state: blocked=55 finished=37 ready-or-running=8
200568 [INFO] Builder state: blocked=55 finished=38 ready-or-running=7 [org.geoserver.extension:gs-grib org.geoserver.extension:gs-teradata org.geoserver.extension:gs-printing org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
201993 [INFO] Builder state: blocked=55 finished=39 ready-or-running=6 [org.geoserver.extension:gs-grib org.geoserver.extension:gs-teradata org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
202183 [INFO] Builder state: blocked=55 finished=40 ready-or-running=5 [org.geoserver.extension:gs-grib org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
203917 [INFO] Builder state: blocked=55 finished=41 ready-or-running=4 [org.geoserver:gs-wcs2_0 org.geoserver:gs-wfs org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
205539 [INFO] Builder state: blocked=54 finished=42 ready-or-running=4 [org.geoserver:gs-wfs org.geoserver.extension:gs-wcs2_0-eo-core org.geoserver:gs-restconfig org.geoserver.security:gs-security-tests]
207482 [INFO] Builder state: blocked=45 finished=43 ready-or-running=12
210406 [INFO] Builder state: blocked=43 finished=44 ready-or-running=13
211602 [INFO] Builder state: blocked=38 finished=45 ready-or-running=17
232568 [INFO] Builder state: blocked=38 finished=46 ready-or-running=16
233239 [INFO] Builder state: blocked=37 finished=47 ready-or-running=16
233768 [INFO] Builder state: blocked=37 finished=48 ready-or-running=15
235338 [INFO] Builder state: blocked=37 finished=49 ready-or-running=14
245546 [INFO] Builder state: blocked=37 finished=50 ready-or-running=13
246113 [INFO] Builder state: blocked=37 finished=51 ready-or-running=12
252089 [INFO] Builder state: blocked=37 finished=52 ready-or-running=11
253813 [INFO] Builder state: blocked=37 finished=53 ready-or-running=10
255022 [INFO] Builder state: blocked=37 finished=54 ready-or-running=9
257966 [INFO] Builder state: blocked=37 finished=55 ready-or-running=8
258552 [INFO] Builder state: blocked=37 finished=56 ready-or-running=7 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.web:gs-web-rest org.geoserver.extension:gs-web-wcs2_0-eo org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
262807 [INFO] Builder state: blocked=37 finished=57 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.extension:gs-web-wcs2_0-eo org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
266017 [INFO] Builder state: blocked=37 finished=58 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-excel org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
266249 [INFO] Builder state: blocked=37 finished=59 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-sample-data-access-test org.geoserver.extension:gs-wps-core]
267150 [INFO] Builder state: blocked=37 finished=60 ready-or-running=3 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-wps-core]
282823 [INFO] Builder state: blocked=34 finished=61 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-dxf-wps org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.extension:gs-ogr-wps]
290936 [INFO] Builder state: blocked=34 finished=62 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-core org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.extension:gs-ogr-wps]
292186 [INFO] Builder state: blocked=31 finished=63 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.extension:gs-security org.geoserver.web:gs-web-sec-jdbc org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
292651 [INFO] Builder state: blocked=29 finished=64 ready-or-running=7 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-web-sec org.geoserver.extension:gs-ogr-wps]
293199 [INFO] Builder state: blocked=29 finished=65 ready-or-running=6 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.extension:gs-wps-cluster-hazelcast org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
293995 [INFO] Builder state: blocked=29 finished=66 ready-or-running=5 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.web:gs-web-sec-ldap org.geoserver.extension:gs-ogr-wps]
295786 [INFO] Builder state: blocked=29 finished=67 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-sec-cas org.geoserver.web:gs-web-sec-ldap]
298512 [INFO] Builder state: blocked=28 finished=68 ready-or-running=4 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.security:gs-web-sec-cas org.geoserver.web:gs-web-sec-ldap]
300423 [INFO] Builder state: blocked=28 finished=69 ready-or-running=3 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc org.geoserver.web:gs-web-sec-ldap]
302346 [INFO] Builder state: blocked=28 finished=70 ready-or-running=2 [org.geoserver:gs-wms org.geoserver.web:gs-web-sec-jdbc]
319821 [INFO] Builder state: blocked=16 finished=71 ready-or-running=13
320386 [INFO] Builder state: blocked=15 finished=72 ready-or-running=13
348110 [INFO] Builder state: blocked=15 finished=73 ready-or-running=12
353240 [INFO] Builder state: blocked=12 finished=74 ready-or-running=14
356878 [INFO] Builder state: blocked=11 finished=75 ready-or-running=14
357299 [INFO] Builder state: blocked=11 finished=76 ready-or-running=13
364581 [INFO] Builder state: blocked=10 finished=77 ready-or-running=13
371091 [INFO] Builder state: blocked=10 finished=78 ready-or-running=12
380418 [INFO] Builder state: blocked=9 finished=79 ready-or-running=12
389545 [INFO] Builder state: blocked=9 finished=80 ready-or-running=11
395658 [INFO] Builder state: blocked=9 finished=81 ready-or-running=10
396566 [INFO] Builder state: blocked=9 finished=82 ready-or-running=9
406503 [INFO] Builder state: blocked=6 finished=83 ready-or-running=11
409425 [INFO] Builder state: blocked=6 finished=84 ready-or-running=10
409602 [INFO] Builder state: blocked=6 finished=85 ready-or-running=9
410405 [INFO] Builder state: blocked=6 finished=86 ready-or-running=8
417603 [INFO] Builder state: blocked=6 finished=87 ready-or-running=7 [org.geoserver.importer:gs-importer-web org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-imagemap]
421548 [INFO] Builder state: blocked=6 finished=88 ready-or-running=6 [org.geoserver.importer:gs-importer-web org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
426601 [INFO] Builder state: blocked=6 finished=89 ready-or-running=5 [org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver.importer:gs-importer-bdb org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
426636 [INFO] Builder state: blocked=6 finished=90 ready-or-running=4 [org.geoserver.importer:gs-importer-rest org.geoserver.extension:gs-netcdf-out org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
430242 [INFO] Builder state: blocked=6 finished=91 ready-or-running=3 [org.geoserver.importer:gs-importer-rest org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
443521 [INFO] Builder state: blocked=6 finished=92 ready-or-running=2 [org.geoserver:gs-gwc org.geoserver.extension:gs-app-schema-test]
456022 [INFO] Builder state: blocked=2 finished=93 ready-or-running=5 [org.geoserver.extension:gs-ysld org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-inspire]
470715 [INFO] Builder state: blocked=2 finished=94 ready-or-running=4 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-app-schema-test org.geoserver.extension:gs-inspire]
475546 [INFO] Builder state: blocked=2 finished=95 ready-or-running=3 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-monitor-core org.geoserver.extension:gs-inspire]
478682 [INFO] Builder state: blocked=2 finished=96 ready-or-running=2 [org.geoserver.web:gs-web-gwc org.geoserver.extension:gs-inspire]
484459 [INFO] Builder state: blocked=1 finished=97 ready-or-running=2 [org.geoserver.web:gs-web-gwc org.geoserver:gs-release]
485380 [INFO] Builder state: blocked=1 finished=98 ready-or-running=1 [org.geoserver.web:gs-web-gwc]
491967 [INFO] Builder state: blocked=0 finished=99 ready-or-running=1
496308 [INFO] Builder state: blocked=0 finished=100 ready-or-running=0

The output starts with a time in milliseconds from the begin of the build, and then info about how many are runnable or running (and I was using intellij on the side, so times are a little longer than usual).
A new message is generated when a module end building.
Unfortunately the output gives info only when there are less runnables than one would like to, but generally speaking, I see the
my CPU (ryzen 1700x, 8 cores) “suffer” (being under-utilized) when the number of running processes goes below 5 or 6.

I’ve highlighted a few sections that show a criticality:

When gs-main is reached, there is a 50 seconds “stop the world” waiting for it to finish (there are a couple of security tests in there that are dead slow)

The gs-wfs starts building (cannot be seen, but it’s there) and the other parallel jobs finish before it finishes, bringing down the concurrent jobs to 4, and then after it lots of parallelization again

After gs-wfs the other beast is gs-wms, we have to wait for it to finish building before another large parallelization happens

App-schema starts building in that phase where there are lots of modules active and finishes almost at the end (parallel builds are not fully reproducable, I often see it finishing last) … my hope is that by splitting it it would start running before and leave more breathing room towards the end for the CPUs to go higher freq
I’m focusing on app-schema tests because it seems the easiest to take apart and in common builds I often see it running towards the end, but major benefits would be reaped by having gs-main build faster, and also having gs-wms not depend on gs-wfs (GetFeatureInfo does GML encoding unfortunately, not sure there are other dependencies… maybe we could have a gs-wms-info module that depends on a simpler gs-wms and gs-wfs… something to try out) so that gs-gwc can start its build sooner. App-schema-test would still depend on the gs-wms-info one, since it’s doing GetFeatureInfo on complex features.

gs-gwc is another module with very slow tests (the is at least one repeating the data dir setup, instead of sharing it) and in a critical position, that could use some work.

After writing this I thought to also check a -T4 build, since most machines are still stuck on 4 cores (including our build servers).
Long story short, a -T4 build is almost fully loaded all the time, and the above changes to app-schema would likely do little to speed it up, that build would likely
be made faster only by focusing on slow tests and making them faster.
I’ll have a look at chasing them down in a sequential build, and follow up.

Cheers
Andrea

–

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

Andrea_Aime4 · November 11, 2018, 10:26am

Hi Gabriel,
for completeness sake, I have tried using a memory FS to store all the data directories on Travis too, commit here:
https://github.com/geoserver/geoserver/pull/3228/commits/1c43f9615a4389638e3fd4e319c76bc8314e57a1

Unfortunately it did not make a difference, the build seems to be CPU bound on the build server too

Cheers
Andrea

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Gabriel_Roldan2 · November 13, 2018, 2:13pm

Hey,
yeah it makes sense since the tests are obviously way less IO bound than geogig’s.
Sorry for the useless suggestion and the accompanied waste of time.

Gabriel.

···

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

Andrea_Aime4 · November 13, 2018, 2:19pm

Hi Gabriel,
no problem, given all the data dir setups it was indeed a good one to investigate anyways

Cheers
Andrea

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.