[Geoserver-devel] Build server: no space left on device

Andrea_Aime4 · June 8, 2021, 8:42am

Hi all,
I though we had plenty of space to separate the build maven repos? And yet,
we’re getting this:

https://build.geoserver.org/job/geoserver-main-nightly/742/display/redirect?page=changes

a.nio.file.FileSystemException: <https://build.geoserver.org/job/geoserver-main-nightly/742/artifact/src/target/release/geoserver-2.20-SNAPSHOT-bin.zip> → /var/lib/jenkins/jobs/geoserver-main-nightly/builds/742/archive/src/target/release/geoserver-2.20-SNAPSHOT-bin.zip: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:253)
at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:581)
at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
at java.nio.file.Files.copy(Files.java:1274)

Hum

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Torben_Barsballe1 · June 8, 2021, 3:28pm

Yeah, there was 80 G free when I last checked in April, which should have been enough, but looking at disk usage I can indeed confirm there is no space left.

I’m digging into exactly where all the space is being used right now…

Can you give me the list of exactly which jobs you modified to use their own maven repo? How many new maven repos were added for this? And where those respective maven repos are?

Cheers,
Torben

On Tue, Jun 8, 2021 at 1:43 AM Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi all,
I though we had plenty of space to separate the build maven repos? And yet,
we’re getting this:

https://build.geoserver.org/job/geoserver-main-nightly/742/display/redirect?page=changes

a.nio.file.FileSystemException: <https://build.geoserver.org/job/geoserver-main-nightly/742/artifact/src/target/release/geoserver-2.20-SNAPSHOT-bin.zip> → /var/lib/jenkins/jobs/geoserver-main-nightly/builds/742/archive/src/target/release/geoserver-2.20-SNAPSHOT-bin.zip: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:253)
at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:581)
at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
at java.nio.file.Files.copy(Files.java:1274)

Hum

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Andrea_Aime4 · June 8, 2021, 3:35pm

One per branch and per GT/GS version. At the beginning I added them only in the GeoTools jobs,
using the same approach as the JDK11 builds (passing the variable to a local m2 folder),
but then Jody wanted to crack up the parallelism on the GeoServer builds too, builds started failing there,
and I did the same on the GeoServer jobs as well (each using a local m2 folder)

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Torben_Barsballe1 · June 8, 2021, 5:25pm

(See “[Geoserver-devel] Build server: no space left on device” for context)

I’ve got some good news - the extra maven repos are not the issue. Looking at geoserver-main, m2 is just 1.2G; for geotools-main m2 is just under 350M.

The actual culprit is the CITE tests, specifically the docker environment, which consumes 80G. Given we had around 40G free in April, I expect this is something that has grown steadily over time, as orphaned docker images tend to do.

I ran a docker system prune, which freed up 75G, so I think if we set this to run every month or so, that should stop this problem from coming back.
I’ve rerun the CITE tests to verify that this prune hasn’t broken anything - some jobs were already failing so I can only really confirm the same ones are still passing/failing). If anyone more familiar with the new cite tests can do a sanity check of this as well, that would be appreciated.

I’ll set up a job to automatically prune the docker system once a month - I think that’ll probably also need to rerun teamengine-docker to rebuild the image.

I also noticed something else while searching for the cause of the disk usage - there are many (over 400) small (~4.5 M) folders in /var/lib/jenkins/workspace of the form “cite-runner_ws-cleanup_1618427258136”. These appear to be dynamically generated and are not automatically cleaned up by anything. If someone who is familiar with those jobs could look into some sort of automated cleanup process for these, that would also be helpful.

Cheers,
Torben

On Tue, Jun 8, 2021 at 8:35 AM Andrea Aime <andrea.aime@anonymised.com> wrote:

On Tue, Jun 8, 2021 at 5:28 PM Torben Barsballe <torbenbarsballe@anonymised.com> wrote:

Can you give me the list of exactly which jobs you modified to use their own maven repo? How many new maven repos were added for this? And where those respective maven repos are?

One per branch and per GT/GS version. At the beginning I added them only in the GeoTools jobs,
using the same approach as the JDK11 builds (passing the variable to a local m2 folder),
but then Jody wanted to crack up the parallelism on the GeoServer builds too, builds started failing there,
and I did the same on the GeoServer jobs as well (each using a local m2 folder)

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Torben_Barsballe1 · June 8, 2021, 6:01pm

The docker cleanup job now exists, and will run on the 1st of each month: teamengine-docker-cleanup
Note: I’ve named it as such mostly so it gets sorted adjacent the teamengine docker job, since that is our only docker job; it’s not technically related to that job, this is just for ease of use / organization purposes.

Cheers,
Torben

On Tue, Jun 8, 2021 at 10:25 AM Torben Barsballe <torbenbarsballe@anonymised.com> wrote:

(See “[Geoserver-devel] Build server: no space left on device” for context)

I’ve got some good news - the extra maven repos are not the issue. Looking at geoserver-main, m2 is just 1.2G; for geotools-main m2 is just under 350M.

The actual culprit is the CITE tests, specifically the docker environment, which consumes 80G. Given we had around 40G free in April, I expect this is something that has grown steadily over time, as orphaned docker images tend to do.

I ran a docker system prune, which freed up 75G, so I think if we set this to run every month or so, that should stop this problem from coming back.
I’ve rerun the CITE tests to verify that this prune hasn’t broken anything - some jobs were already failing so I can only really confirm the same ones are still passing/failing). If anyone more familiar with the new cite tests can do a sanity check of this as well, that would be appreciated.

I’ll set up a job to automatically prune the docker system once a month - I think that’ll probably also need to rerun teamengine-docker to rebuild the image.

I also noticed something else while searching for the cause of the disk usage - there are many (over 400) small (~4.5 M) folders in /var/lib/jenkins/workspace of the form “cite-runner_ws-cleanup_1618427258136”. These appear to be dynamically generated and are not automatically cleaned up by anything. If someone who is familiar with those jobs could look into some sort of automated cleanup process for these, that would also be helpful.

Cheers,
Torben

On Tue, Jun 8, 2021 at 8:35 AM Andrea Aime <andrea.aime@anonymised.com.> wrote:

On Tue, Jun 8, 2021 at 5:28 PM Torben Barsballe <torbenbarsballe@anonymised.com> wrote:

Can you give me the list of exactly which jobs you modified to use their own maven repo? How many new maven repos were added for this? And where those respective maven repos are?

One per branch and per GT/GS version. At the beginning I added them only in the GeoTools jobs,
using the same approach as the JDK11 builds (passing the variable to a local m2 folder),
but then Jody wanted to crack up the parallelism on the GeoServer builds too, builds started failing there,
and I did the same on the GeoServer jobs as well (each using a local m2 folder)

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Alessandro_Parma · June 9, 2021, 9:10am

Awesome, thanks Torben!

···

Regards,

Alessandro Parma

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Alessandro Parma
DevOps Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.