[Geoserver-devel] too many open files building with java 11

GeoServer GeoServer Developer
1

In anticipation of dropping java 8 support I switched to Java 11 for my daily driver (local builds and such). In the last week I have been getting “too many open files” when building?

Env:

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /usr/local/Cellar/maven/3.8.4/libexec
Java version: 11.0.15, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
Default locale: en_CA, platform encoding: UTF-8
OS name: “mac os x”, version: “12.6”, arch: “x86_64”, family: “mac”

Has anyone else seen this kind of error? Should I report to Jira or is something just wrong on me end (example using macOS :smiley: ).

···

–
Jody Garnett

2

I’m not seeing this, but when I checked ulimit, it says “unlimited”.
I’ve checked the build server, it also says the same, unlimited.

Out Github actions with JDK 11 are not failing and they don’t change the ulimit of the OS.
This ticket is discussing the defaults, the comment I’ve linked to seems to imply
the default is 10635… a PR could be used to modify the build script adding a check on
ulimit to make sure.

Cheers
Andrea

···

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

3

I was able to build by changing my usual build “rebuild” from:

mvn clean install -Prelease,communityRelease -DskipTests -T1C -nsu -fae -ntp

To use less threads:
mvn clean install -Prelease,communityRelease -DskipTests -T4 -nsu -fae -ntp

Searching the internet asks me to check errorProne but I do not think that is used unless doing a QA build.

···

–
Jody Garnett

4

Check your ulimit you may want to increase using eg. ulimit -n 10240

Op do 29 sep. 2022 20:49 schreef Jody Garnett <jody.garnett@anonymised.com>:

I was able to build by changing my usual build “rebuild” from:

mvn clean install -Prelease,communityRelease -DskipTests -T1C -nsu -fae -ntp

To use less threads:
mvn clean install -Prelease,communityRelease -DskipTests -T4 -nsu -fae -ntp

Searching the internet asks me to check errorProne but I do not think that is used unless doing a QA build.

–
Jody Garnett

On Thu, 29 Sept 2022 at 00:55, Andrea Aime <andrea.aime@anonymised.com> wrote:

I’m not seeing this, but when I checked ulimit, it says “unlimited”.
I’ve checked the build server, it also says the same, unlimited.

Out Github actions with JDK 11 are not failing and they don’t change the ulimit of the OS.
This ticket is discussing the defaults, the comment I’ve linked to seems to imply
the default is 10635… a PR could be used to modify the build script adding a check on
ulimit to make sure.

Cheers
Andrea

On Thu, Sep 29, 2022 at 1:56 AM Jody Garnett <jody.garnett@anonymised.com> wrote:

In anticipation of dropping java 8 support I switched to Java 11 for my daily driver (local builds and such). In the last week I have been getting “too many open files” when building?

Env:

Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /usr/local/Cellar/maven/3.8.4/libexec
Java version: 11.0.15, vendor: Eclipse Adoptium, runtime: /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
Default locale: en_CA, platform encoding: UTF-8
OS name: “mac os x”, version: “12.6”, arch: “x86_64”, family: “mac”

Has anyone else seen this kind of error? Should I report to Jira or is something just wrong on me end (example using macOS :smiley: ).

–
Jody Garnett

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

5

Trying various values, with 65535 being a common recommendation, and still encountering the error (even with -T3).

Double check: Latest JDK 11.0.16.1, and maven 3.8.6 …

Going to try changing how the compiler is called:

  1. Using -Perrorprone (because why not) and everything is fine (well errorprone failures in community modules but that is expected)

  2. Updating to maven-compile-plugin 3.10.1 (because why not) and everything is fine:

Resulting PR: https://github.com/geoserver/geoserver/pull/6217

···

–
Jody Garnett