[Geoserver-devel] clean up tmp dir after tests

GeoServer GeoServer Developer
1

Hello,

sorry for the cross-posting but this is about geosever and geotools builds in ares.

The build process leaves a lot, literally thousands, of files and directories in the temp directory.

I cam across this looking for the reason some geogig temp directories are also left over where they shouldn’t because we’re supposed to be using junit’s TemporaryFolder everywhere, and turns out we missed a couple.

So, the “right way” of solving this inconvenience would be to upgrade all geotools and geoserver unit/integration tests to use a TemporaryFolder rule. Now, that’d be a pretty monumental task.

But I came up with this other idea: we can clean up after the build in the shell script that’s executed, and isolate the temporary resources of each projects on their own temp folder with the -Djava.io.tmpdir system property.

In a nutshell:

  • have /tmp/getotools- and /tmp/geoserver- directories
  • run maven with mvn -Djava.io.tmpdir=/tmp/geotools|geoserver-
  • run rm -rf /tmp/geotools|geoserver-/* in the post-build shell script

and voilá.

Moreover, ares doesn’t have it’s /tmp in its own partition, and there’s a tmpfs (i.e. memory) mounted at /mnt/memtmp which we use to speed up the geogig build (that is, running tests that do IO in a tmpfs). Would you guys be interested in doing that too?

Best regards,
Gabriel

···

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

2

If you do this with maven (which can include ant scripts directly tasks like this) then it can still be cross platform. I wonder if it is possible to set the java temp directory provided to the test runner?

···

On 5 August 2016 at 12:56, Gabriel Roldan <groldan@anonymised.com> wrote:

Hello,

sorry for the cross-posting but this is about geosever and geotools builds in ares.

The build process leaves a lot, literally thousands, of files and directories in the temp directory.

I cam across this looking for the reason some geogig temp directories are also left over where they shouldn’t because we’re supposed to be using junit’s TemporaryFolder everywhere, and turns out we missed a couple.

So, the “right way” of solving this inconvenience would be to upgrade all geotools and geoserver unit/integration tests to use a TemporaryFolder rule. Now, that’d be a pretty monumental task.

But I came up with this other idea: we can clean up after the build in the shell script that’s executed, and isolate the temporary resources of each projects on their own temp folder with the -Djava.io.tmpdir system property.

In a nutshell:

  • have /tmp/getotools- and /tmp/geoserver- directories
  • run maven with mvn -Djava.io.tmpdir=/tmp/geotools|geoserver-
  • run rm -rf /tmp/geotools|geoserver-/* in the post-build shell script

and voilá.

Moreover, ares doesn’t have it’s /tmp in its own partition, and there’s a tmpfs (i.e. memory) mounted at /mnt/memtmp which we use to speed up the geogig build (that is, running tests that do IO in a tmpfs). Would you guys be interested in doing that too?

Best regards,
Gabriel

–

Gabriel Roldán
Software Developer | Boundless
groldan@anonymised.com
@boundlessgeo

GeoTools-Devel mailing list
GeoTools-Devel@anonymised.com66…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

–
Jody Garnett

3

On Fri, Aug 5, 2016 at 9:56 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

In a nutshell:
- have /tmp/getotools-<version> and /tmp/geoserver-<version> directories
- run maven with mvn -Djava.io.tmpdir=/tmp/geotools|geoserver-<version>
- run rm -rf /tmp/geotools|geoserver-<version>/* in the post-build shell
script

Works for me.

and voilá.

Moreover, ares doesn't have it's /tmp in its own partition, and there's a
tmpfs (i.e. memory) mounted at /mnt/memtmp which we use to speed up the
geogig build (that is, running tests that do IO in a tmpfs). Would you guys
be interested in doing that too?

It might provide some benefit, but not all that much in geoserver where
most test resources (e.g., the data dirs) are created
inside "<module>/target" (and I believe that is valid for many, but not
all, geotools modules as well).
I guess that could be made parametric too to get a speedup, I'm just not
sure how well isolated the
tests would be for parallel builds (I believe they are using some sort of
UUID like mechanism to name them, but I might be wrong)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

4

On Sat, Aug 6, 2016 at 5:47 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Fri, Aug 5, 2016 at 9:56 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

In a nutshell:
- have /tmp/getotools-<version> and /tmp/geoserver-<version> directories
- run maven with mvn -Djava.io.tmpdir=/tmp/geotools|geoserver-<version>
- run rm -rf /tmp/geotools|geoserver-<version>/* in the post-build shell
script

Works for me.

ok, the last geoserver builds ran fine with -Djava.io.tmpdir=/tmp/geoserver
and cleaned up after by adding rm -rf /tmp/geoserver/* in the post-build
shell script.
However, I ssh'ed again to ares today and the /tmp/geoserver directory is
gone. Checked I didn't accidentally add `rm -rf /tmp/geoserver ` instead of
`rm -rf /tmp/geoserver/*` and no, I didn't. So wonder what's going on.

By the other side, found out the geotools builds are currently being run
with -Djava.io.tmpdir=/var/lib/jenkins/tmp
There are currently 49814 files/directories in that location.
Does anybody know why they're being run with that tmp directory, and what
should we do? can just leave it as is and add `rm -rf
/var/lib/jenkins/tmp/*` to the post script

and voilá.

Moreover, ares doesn't have it's /tmp in its own partition, and there's a
tmpfs (i.e. memory) mounted at /mnt/memtmp which we use to speed up the
geogig build (that is, running tests that do IO in a tmpfs). Would you guys
be interested in doing that too?

It might provide some benefit, but not all that much in geoserver where
most test resources (e.g., the data dirs) are created
inside "<module>/target" (and I believe that is valid for many, but not
all, geotools modules as well).
I guess that could be made parametric too to get a speedup, I'm just not
sure how well isolated the
tests would be for parallel builds (I believe they are using some sort of
UUID like mechanism to name them, but I might be wrong)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

--

Gabriel Roldán
Software Developer | Boundless <http://boundlessgeo.com/&gt;
groldan@anonymised.com
@boundlessgeo <http://twitter.com/boundlessgeo/&gt;