[Geoserver-devel] GeoServer builds and runs on JDK11

GeoServer GeoServer Developer
1

Hi,
you have seen what this is about from the subject… before delving into the details, let’s have a look at the results:

[INFO] Ysld GeoServer Plugin … SUCCESS [ 15.820 s]
[INFO] MongoDB Data Store … SUCCESS [ 0.803 s]
[INFO] REST SLD service … SUCCESS [03:59 min]
[INFO] GeoFence security integration … SUCCESS [01:19 min]
[INFO] GeoFence Server … SUCCESS [01:55 min]
[INFO] GeoServer Release Module 2.15-SNAPSHOT … SUCCESS [ 0.820 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:04 min (Wall Clock)
[INFO] Finished at: 2018-10-14T16:04:18+02:00
[INFO] ------------------------------------------------------------------------
~/devel/git-gs/src (jdk11_build) $ mvn -version
Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T20:49:05+01:00)
Maven home: /home/aaime/apps/apache-maven-3.5.3
Java version: 11, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk-11
Default locale: it_IT, platform encoding: UTF-8
OS name: “linux”, version: “4.15.0-34-generic”, arch: “amd64”, family: “unix”

(forget about that vendor indication, it’s actually OpenJDK):

~/devel/git-gs/src (jdk11_build) $ java -version
openjdk version “11” 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

image.png

In order for the above to happen two PRs need to be applied, in addition to the ones already presented
for imageio-ext and geotools:

but the number of modified files is overall not big, nothing particularly fancy.

Bits worth noticing:

  • HashMap iteration order changed once more, a few tests needed to be amended
  • XML encoding of numbers and CDATA changed (different precision, different spacing around CDATA), some tests had to be amended
  • Changing the size of a thread pool at runtime incurs into stricter checks than before
  • GeoServerExtensions integration with SPI broke, I fell down on the GeoTools FactoryRegistry code, not sure if what was done is the best way to handle it, I’m open to suggestions:
    https://github.com/geoserver/geoserver/pull/3182/files#diff-8414b5ac2b07c761343e3d61d80cf320R154
  • The security LDAP module now builds (before it was trying to access a class that’s truly gone due to a debugging statement in the embedded LDAP server being used), but tests do not run (get skipped) because now the embedded server won’t start… even all I did was to bump it a couple of bugfix versions (bumping just one had the same result). That, and the newer version of the server is still a milestone and the spring ldap integration is declared not to work with it. Think we need someone that understands LDAP to make it actually run tests again.
  • Commons-lang3 had to be upgraded to the latest version, as some utilities triggered Java version number parsing and failed. No biggie, but GWC had to be upgraded from commons-lang 2, which makes up most of the file changes in the PR
  • Hazelcast has been upgraded to the latest version, which is a beta, but also the first run supposed to be running on JDK11. Still triggers some warnings, checking their issue tracker it seems at least some are being worked on before final release.
    Going back to the performance topic the build times are a joke, using JDK8 and a “-T8” build I normally build on 7:30 to 8:00, with JDK11 it takes around 14 minutes!

Are we screwed? We’ll see, in the meantime I did a quick and silly benchmark against topp:states:

ab -n 3600 -c 16 “http://localhost:8080/geoserver/topp/wms?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=topp%3Astates&exceptions=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&STYLES=&WIDTH=768&HEIGHT=330&BBOX=-130.60546875%2C23.37890625%2C-63.10546875%2C52.3828125”

Concurrency Level: 16
Time taken for tests: 8.729 seconds
Complete requests: 3600
Failed requests: 0
Total transferred: 251305200 bytes
HTML transferred: 250491600 bytes
Requests per second: 412.44 [#/sec] (mean)
Time per request: 38.794 [ms] (mean)
Time per request: 2.425 [ms] (mean, across all concurrent requests)
Transfer rate: 28116.37 [Kbytes/sec] received

The performance on JDK8 + marlin is, by memory, the same (it’s late and I don’t want to rebuild everything in JDK 8 to confirm.
Also note the GeoServer status page screenshot, you’ll notice that JDK11 is using its own embedded Marlin version (as you probably
know, it has become the default renderer in JDK since version 9).
So, it seems something is up with the build but maybe not on runtime (more checks are needed).

Soo… next steps:

  • I have to look into the GeoTools build and make it mergeable
  • A review on the various PRs would be much appreciated
  • Once the two above are done, I’d suggest to merge and call “step 1” done, so that when the sprint starts we can fork off and concentrate on warnings removal and modularization/re-packaging
    Cheers

Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

2

Nice work.

I had a quick look at the GWC changes. The lang3 and surefire upgrades look like they could go on master now. Also the oracle releases maven repo removal looks like it could go in.

That surefire upgrade is probably worth doing on other projects too.

We probably should find better matchers for the tests that are brittle to order and format. A bit more work now, but less churn when Java moves on again.

Brad

From: Andrea Aime <andrea.aime@anonymised.com…>
Sent: Monday, 15 October 2018 5:39 AM
To: Geoserver-devel geoserver-devel@lists.sourceforge.net
Subject: [Geoserver-devel] GeoServer builds and runs on JDK11

Hi,

you have seen what this is about from the subject… before delving into the details, let’s have a look at the results:

[INFO] Ysld GeoServer Plugin … SUCCESS [ 15.820 s]

[INFO] MongoDB Data Store … SUCCESS [ 0.803 s]

[INFO] REST SLD service … SUCCESS [03:59 min]

[INFO] GeoFence security integration … SUCCESS [01:19 min]

[INFO] GeoFence Server … SUCCESS [01:55 min]

[INFO] GeoServer Release Module 2.15-SNAPSHOT … SUCCESS [ 0.820 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 14:04 min (Wall Clock)

[INFO] Finished at: 2018-10-14T16:04:18+02:00

[INFO] ------------------------------------------------------------------------

~/devel/git-gs/src (jdk11_build) $ mvn -version

Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T20:49:05+01:00)

Maven home: /home/aaime/apps/apache-maven-3.5.3

Java version: 11, vendor: Oracle Corporation

Java home: /usr/lib/jvm/jdk-11

Default locale: it_IT, platform encoding: UTF-8

OS name: “linux”, version: “4.15.0-34-generic”, arch: “amd64”, family: “unix”

(forget about that vendor indication, it’s actually OpenJDK):

~/devel/git-gs/src (jdk11_build) $ java -version

openjdk version “11” 2018-09-25

OpenJDK Runtime Environment 18.9 (build 11+28)

OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

image.png

In order for the above to happen two PRs need to be applied, in addition to the ones already presented

for imageio-ext and geotools:

As you can see by the numbers of commits in those two, it has not been as easy as with GeoTools and friends,

but the number of modified files is overall not big, nothing particularly fancy.

Bits worth noticing:

  • HashMap iteration order changed once more, a few tests needed to be amended
  • XML encoding of numbers and CDATA changed (different precision, different spacing around CDATA), some tests had to be amended
  • Changing the size of a thread pool at runtime incurs into stricter checks than before
  • GeoServerExtensions integration with SPI broke, I fell down on the GeoTools FactoryRegistry code, not sure if what was done is the best way to handle it, I’m open to suggestions:
    https://github.com/geoserver/geoserver/pull/3182/files#diff-8414b5ac2b07c761343e3d61d80cf320R154
  • The security LDAP module now builds (before it was trying to access a class that’s truly gone due to a debugging statement in the embedded LDAP server being used), but tests do not run (get skipped) because now the embedded server won’t start… even all I did was to bump it a couple of bugfix versions (bumping just one had the same result). That, and the newer version of the server is still a milestone and the spring ldap integration is declared not to work with it. Think we need someone that understands LDAP to make it actually run tests again.
  • Commons-lang3 had to be upgraded to the latest version, as some utilities triggered Java version number parsing and failed. No biggie, but GWC had to be upgraded from commons-lang 2, which makes up most of the file changes in the PR
  • Hazelcast has been upgraded to the latest version, which is a beta, but also the first run supposed to be running on JDK11. Still triggers some warnings, checking their issue tracker it seems at least some are being worked on before final release.

Going back to the performance topic the build times are a joke, using JDK8 and a “-T8” build I normally build on 7:30 to 8:00, with JDK11 it takes around 14 minutes!

Are we screwed? We’ll see, in the meantime I did a quick and silly benchmark against topp:states:

ab -n 3600 -c 16 “http://localhost:8080/geoserver/topp/wms?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=topp%3Astates&exceptions=application%2Fvnd.ogc.se_inimage&SRS=EPSG%3A4326&STYLES=&WIDTH=768&HEIGHT=330&BBOX=-130.60546875%2C23.37890625%2C-63.10546875%2C52.3828125”

Concurrency Level: 16

Time taken for tests: 8.729 seconds

Complete requests: 3600

Failed requests: 0

Total transferred: 251305200 bytes

HTML transferred: 250491600 bytes

Requests per second: 412.44 [#/sec] (mean)

Time per request: 38.794 [ms] (mean)

Time per request: 2.425 [ms] (mean, across all concurrent requests)

Transfer rate: 28116.37 [Kbytes/sec] received

The performance on JDK8 + marlin is, by memory, the same (it’s late and I don’t want to rebuild everything in JDK 8 to confirm.

Also note the GeoServer status page screenshot, you’ll notice that JDK11 is using its own embedded Marlin version (as you probably

know, it has become the default renderer in JDK since version 9).

So, it seems something is up with the build but maybe not on runtime (more checks are needed).

Soo… next steps:

  • I have to look into the GeoTools build and make it mergeable
  • A review on the various PRs would be much appreciated
  • Once the two above are done, I’d suggest to merge and call “step 1” done, so that when the sprint starts we can fork off and concentrate on warnings removal and modularization/re-packaging

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

3

Nice work.

I had a quick look at the GWC changes. The lang3 and surefire upgrades look like they could go on master now. Also the oracle releases maven repo removal looks like it could go in.

I actually hope to get that merged before Monday. The sprint plan is, after all, to get the results of the sprint merged before the end of next week, and to get merges going
during the week as the various steps build and run correctly (it’s the dev series, in fact of such large changes we can handle a bit of instability, as an exception to the norm).

That surefire upgrade is probably worth doing on other projects too.

Yeah, wanted to, but I think I’ve tried it on GeoTools and tests started failing. Did not have time to look into it, any takers?

We probably should find better matchers for the tests that are brittle to order and format. A bit more work now, but less churn when Java moves on again.

Good idea, I’m afraid there might still be a lot of tests still matching doubles as strings, and it’s a manual conversion work. Again, any takers welcomed :slight_smile:

Cheers
Andrea

(attachments)

image002.jpg

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

4

I tried the surefire upgrade and it worked fine locally (with JDK8).

Put up https://github.com/geotools/geotools/pull/2102 to see what Travis and AppVeyor say.

Brad

5

Going to dig through this more later, but wanted to thank you for the work and amazing result.

As for build times, is the compiler slower - or is it surefire being slower?

(attachments)

image.png

···

–
Jody Garnett

6

See previous mails, the short answer is: both
Interestingly, a running GeoServer does not seem slower on a quick test…

Cheers
Andrea

···

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.