[Geoserver-devel] ldap failures on build.geoserver.org

I created three build categories to make better use of the build server:

  • build a - master
  • build b - 2.17.x (stable)
  • build c - 2.16.x (maintenance)

We can rotate trough the categories as each release is made.

Spinning up new nodes has expected pain:

  • long initial build
  • occasional empty pom.xml (extension/gs-libjpeg-turbo/maven-metadata-local.xml in this case).

However there are two failures I have not seen before - master and 2.17.x both fail with ldap failures, are these tests especially brittle? I have not had trouble with them before.


Jody Garnett

As far as I remember these tests start an embedded LDAP server which is opening a port.
There is an EmbeddedServer class in the gs-sec-ldap module, but it does not appear to be used (a breakpoint never stops during the test runs)
I see however a @anonymised.com annotation, see https://directory.apache.org/apacheds/advanced-ug/7-embedding-apacheds.html
Supposedly it’s starting a LDAP server on a randomized port but… who knows, maybe the description is not a match for the code we have, it
might be older?

I’m cc’ing Fernando, he setup the tests as they are today, surely knows more than me.
Jody, do you have an example of test failure?

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Ah ha, check that fixed port number:

https://github.com/geoserver/geoserver/blob/c6ec068909cb552333d2a5ae0ea314ca37218b7b/src/security/ldap/src/test/java/org/geoserver/security/ldap/LDAPAuthenticationProviderTest.java#L44

Perhaps the tests can be changed to use a random port, not sure how hard it would be.

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Master is now building, its failure was:

<b>[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.491 s <<< FAILURE! - in org.geoserver.security.ldap.LDAPAuthenticationProviderTest$LDAPAuthenticationProviderData3Test
[ERROR] org.geoserver.security.ldap.LDAPAuthenticationProviderTest$LDAPAuthenticationProviderData3Test  Time elapsed: 3.491 s  <<< ERROR!
java.lang.Exception: 
ERR_317 Schema load failed : Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04317 The created MatchingRule must refers to an existing SYNTAX element
Error : ERR_04306 Cannot find a Syntax object 1.3.6.1.4.1.1466.115.121.1.26 while building cross-references for the krb5RealmName AttributeType.</b> 
···


Jody Garnett

Perhaps add geotools version number to that constant? So each branch uses a fixed port …

public static final int LDAP_SERVER_PORT = 10389 + GeoTools.getVersion();

···


Jody Garnett

Well, that might actually work, at least for the build server :smiley:
Needs a minor change in the next line too:

https://github.com/geoserver/geoserver/blob/c6ec068909cb552333d2a5ae0ea314ca37218b7b/src/security/ldap/src/test/java/org/geoserver/security/ldap/LDAPTestUtils.java#L17

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Because this constant is passed into annotations … it is restricted to things the compiler can manage (thus we reference static methods).

LDAPAuthenticationProviderTest.java:[49,37] element value must be a constant expression
LDAPAuthenticationProviderTest.java:[298,37] element value must be a constant expression
LDAPUserGroupServiceTest.java:[33,33] element value must be a constant expression

Only thing I can think to do is define a static geoserver constant somewhere with we can change across branches :stuck_out_tongue:

Or just change this LDAP_SERVER_PORT each time we branch :frowning:


Jody Garnett

On Sat, 30 May 2020 at 07:05, Andrea Aime <andrea.aime@anonymised.com> wrote:

Well, that might actually work, at least for the build server :smiley:
Needs a minor change in the next line too:

https://github.com/geoserver/geoserver/blob/c6ec068909cb552333d2a5ae0ea314ca37218b7b/src/security/ldap/src/test/java/org/geoserver/security/ldap/LDAPTestUtils.java#L17

Cheers
Andrea

On Sat, May 30, 2020 at 3:43 PM Jody Garnett <jody.garnett@anonymised.com…> wrote:

Perhaps add geotools version number to that constant? So each branch uses a fixed port …

public static final int LDAP_SERVER_PORT = 10389 + GeoTools.getVersion();


Jody Garnett

On Sat, 30 May 2020 at 01:37, Andrea Aime <andrea.aime@anonymised.com> wrote:

Ah ha, check that fixed port number:

https://github.com/geoserver/geoserver/blob/c6ec068909cb552333d2a5ae0ea314ca37218b7b/src/security/ldap/src/test/java/org/geoserver/security/ldap/LDAPAuthenticationProviderTest.java#L44

Perhaps the tests can be changed to use a random port, not sure how hard it would be.

Cheers
Andrea

On Sat, May 30, 2020 at 10:01 AM Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sat, May 30, 2020 at 9:21 AM Jody Garnett <jody.garnett@anonymised.com> wrote:

However there are two failures I have not seen before - master and 2.17.x both fail with ldap failures, are these tests especially brittle? I have not had trouble with them before.

As far as I remember these tests start an embedded LDAP server which is opening a port.
There is an EmbeddedServer class in the gs-sec-ldap module, but it does not appear to be used (a breakpoint never stops during the test runs)
I see however a @anonymised.comeateDS annotation, see https://directory.apache.org/apacheds/advanced-ug/7-embedding-apacheds.html
Supposedly it’s starting a LDAP server on a randomized port but… who knows, maybe the description is not a match for the code we have, it
might be older?

I’m cc’ing Fernando, he setup the tests as they are today, surely knows more than me.
Jody, do you have an example of test failure?

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Jody,
pity, that would have been easy, had it worked.
Another possible approach is not to pass the port to the annotation… I believe it’s going to pick a random one. Then the tests
need to be refactored to discover the port… I cannot find a way to find that out though. Maybe Fernando knows.

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

I did the simple thing for now.

···


Jody Garnett

Right, I can see that:
https://github.com/geoserver/geoserver/commit/b6315fb2ecb81f99ac044b050242c30fbdd74a4c

https://github.com/geoserver/geoserver/commit/af54b06bfad620132372a22dc78bee95df4773c3

If the random port support does not materialize, we’ll have to remember to switch the port numbers in September.

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Here is an update for the release guide, ideally I would like to make a release.xml ant file like GeoTools.

Remaining issues:

  1. geoserver-master-nightly: Compilation failure on gs-wfs3 (fmt plugin is getting an array index out of bounds)
  2. geoserver-2.16.x-java11: Compilation problems with gs-wms (example: “cannot find cannot find symbol: class Resource”) that does not make much sense to me yet
···


Jody Garnett

Okay build is restored, the only odd thing in the setup is geoserver-nightly being in build group a (because it is downstream of geoserver-master and I do not want it to conflict on LDAP port)

···


Jody Garnett

Hi Andrea, Jody

Sorry for the delay checking this, I was dead busy these days.

I did a fast investigation and made a draft commit with a diff for having random ports on LDAP tests. It is based on documentation found on:
https://directory.apache.org/apacheds/advanced-ug/7-embedding-apacheds.html

Commit to check:

https://github.com/fernandor777/geoserver/commit/1ef30e393fada3cc4be610482a304746f1e7a716

If the changes are good for all, I could proceed to create the JIRA ticket (if there is one already created please point it) and the PR.

Thanks.

Regards,

Fernando Mino

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Hi Fernando,
the changes look indeed good to me, please go on!

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.