[Geoserver-devel] Ares build server stuck

GeoServer GeoServer Developer
1

The only “executor” is marked as dead:

http://ares.boundlessgeo.com/jenkins/

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

2

Yikes, server out of disk space. I am on it.

···

On Fri, Mar 21, 2014 at 10:09 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

The only “executor” is marked as dead:

http://ares.boundlessgeo.com/jenkins/

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Learn Graph Databases - Download FREE O’Reilly Book
“Graph Databases” is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

3

I’m pretty sure one of the reasons why it runs out of disk space is that both the geoserver and geotools builds leave a lot of dangling files and directories in /tmp. It usually has thousands of them. I’d be cool if we could do a review of the test cases in both projects to make sure temporary resources get deleted. A good way to do so is to use JUnit4’s TemporaryFolder utility: <https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47>

2c/
Gabriel

···

On Fri, Mar 21, 2014 at 1:34 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Yikes, server out of disk space. I am on it.

Learn Graph Databases - Download FREE O’Reilly Book
“Graph Databases” is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

On Fri, Mar 21, 2014 at 10:09 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

The only “executor” is marked as dead:

http://ares.boundlessgeo.com/jenkins/

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Learn Graph Databases - Download FREE O’Reilly Book
“Graph Databases” is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

4

Back up now.

Looking at what was in /tmp there was about 647M of files there, indeed most from it from the GeoTools / GeoServer build. Only a small percentage of the available disk on the server but +1 on trying to clean that up.

···

On Fri, Mar 21, 2014 at 10:41 AM, Gabriel Roldan <groldan@anonymised.com> wrote:

I’m pretty sure one of the reasons why it runs out of disk space is that both the geoserver and geotools builds leave a lot of dangling files and directories in /tmp. It usually has thousands of them. I’d be cool if we could do a review of the test cases in both projects to make sure temporary resources get deleted. A good way to do so is to use JUnit4’s TemporaryFolder utility: <https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47>

2c/
Gabriel

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

On Fri, Mar 21, 2014 at 1:34 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Yikes, server out of disk space. I am on it.

Learn Graph Databases - Download FREE O’Reilly Book
“Graph Databases” is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

On Fri, Mar 21, 2014 at 10:09 AM, Andrea Aime <andrea.aime@…1268…> wrote:

The only “executor” is marked as dead:

http://ares.boundlessgeo.com/jenkins/

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

Learn Graph Databases - Download FREE O’Reilly Book
“Graph Databases” is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

5

On Fri, Mar 21, 2014 at 5:41 PM, Gabriel Roldan <groldan@anonymised.com>wrote:

I'm pretty sure one of the reasons why it runs out of disk space is that
both the geoserver and geotools builds leave a lot of dangling files and
directories in /tmp. It usually has thousands of them. I'd be cool if we
could do a review of the test cases in both projects to make sure temporary
resources get deleted. A good way to do so is to use JUnit4's
TemporaryFolder utility: <
https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47
>

Our current way of doing things is to have temp files be created under
/target, but not all tests do that unfortunately.

The reason is that it's often helpful to be able to analyze the files after
the test has run, something that the junit
support does not seem to allow

Justin, do you have a list of file names that are getting generated,
hopefully that will make it easier to pinpoint
the modules that are generating them?

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

6

On Fri, Mar 21, 2014 at 2:16 PM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Fri, Mar 21, 2014 at 5:41 PM, Gabriel Roldan <groldan@anonymised.com>wrote:

I'm pretty sure one of the reasons why it runs out of disk space is that
both the geoserver and geotools builds leave a lot of dangling files and
directories in /tmp. It usually has thousands of them. I'd be cool if we
could do a review of the test cases in both projects to make sure temporary
resources get deleted. A good way to do so is to use JUnit4's
TemporaryFolder utility: <
https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47
>

Our current way of doing things is to have temp files be created under
/target, but not all tests do that unfortunately.

The reason is that it's often helpful to be able to analyze the files
after the test has run, something that the junit
support does not seem to allow

I used it with an overridden create() method to create the temp folder
under target/ before just for the same reason. It would probably be a good
idea to use that trick in a geotools subclass. Nice thing about it is you
don't need to write anything in setUp/tearDown, just define the
TemporaryFolder as an instance variable with the @Rule annotation. In any
case, just saying, it could be done any other way too.

Justin, do you have a list of file names that are getting generated,
hopefully that will make it easier to pinpoint
the modules that are generating them?

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--

Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com
@gabrielroldan <https://twitter.com/gabrielroldan&gt;

7

I recall once trying to use -Djava.io.tmpdir to isolate builds and found that some fixtures appear to ignore it; this was some time ago and things might be better now. Furthermore, the first build to write an EPDG database into /tmp wins. Quite unhygienic when multiple builds run on a single host. I was never able to find a satisfactory solution.

Kind regards,
Ben.

On 22/03/14 00:47, Justin Deoliveira wrote:

Back up now.

Looking at what was in /tmp there was about 647M of files there, indeed
most from it from the GeoTools / GeoServer build. Only a small
percentage of the available disk on the server but +1 on trying to clean
that up.

On Fri, Mar 21, 2014 at 10:41 AM, Gabriel Roldan
<groldan@anonymised.com <mailto:groldan@anonymised.com>> wrote:

    I'm pretty sure one of the reasons why it runs out of disk space is
    that both the geoserver and geotools builds leave a lot of dangling
    files and directories in /tmp. It usually has thousands of them. I'd
    be cool if we could do a review of the test cases in both projects
    to make sure temporary resources get deleted. A good way to do so is
    to use JUnit4's TemporaryFolder utility:
    <https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47&gt;

    2c/
    Gabriel

    On Fri, Mar 21, 2014 at 1:34 PM, Justin Deoliveira
    <jdeolive@anonymised.com <mailto:jdeolive@anonymised.com>> wrote:

        Yikes, server out of disk space. I am on it.

        On Fri, Mar 21, 2014 at 10:09 AM, Andrea Aime
        <andrea.aime@anonymised.com
        <mailto:andrea.aime@anonymised.com>> wrote:

            The only "executor" is marked as dead:

            http://ares.boundlessgeo.com/jenkins/

            Cheers
            Andrea

            --
            ==
            Meet us at GEO Business 2014! in London! Visit
            http://goo.gl/fES3aK
            for more information.
            ==

            Ing. Andrea Aime
            @geowolf
            Technical Lead

            GeoSolutions S.A.S.
            Via Poggio alle Viti 1187
            55054 Massarosa (LU)
            Italy
            phone: +39 0584 962313 <tel:%2B39%200584%20962313>
            fax: +39 0584 1660272 <tel:%2B39%200584%201660272>
            mob: +39 339 8844549 <tel:%2B39%20%C2%A0339%208844549>

            http://www.geo-solutions.it
            http://twitter.com/geosolutions_it

            -------------------------------------------------------

            ------------------------------------------------------------------------------
            Learn Graph Databases - Download FREE O'Reilly Book
            "Graph Databases" is the definitive new guide to graph
            databases and their
            applications. Written by three acclaimed leaders in the field,
            this first edition is now available. Download your free book
            today!
            http://p.sf.net/sfu/13534_NeoTech
            _______________________________________________
            Geoserver-devel mailing list
            Geoserver-devel@lists.sourceforge.net
            <mailto:Geoserver-devel@lists.sourceforge.net>
            https://lists.sourceforge.net/lists/listinfo/geoserver-devel

        --
        *Justin Deoliveira*
        Vice President, Engineering | Boundless
        jdeolive@anonymised.com <mailto:jdeolive@anonymised.com>
        @j_deolive <https://twitter.com/j_deolive&gt;

        ------------------------------------------------------------------------------
        Learn Graph Databases - Download FREE O'Reilly Book
        "Graph Databases" is the definitive new guide to graph databases
        and their
        applications. Written by three acclaimed leaders in the field,
        this first edition is now available. Download your free book today!
        http://p.sf.net/sfu/13534_NeoTech
        _______________________________________________
        Geoserver-devel mailing list
        Geoserver-devel@lists.sourceforge.net
        <mailto:Geoserver-devel@lists.sourceforge.net>
        https://lists.sourceforge.net/lists/listinfo/geoserver-devel

    --

    Gabriel Roldán

    Software Developer |Boundless

    groldan@anonymised.com <mailto:groldan@anonymised.com>

    @gabrielroldan <https://twitter.com/gabrielroldan&gt;

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com <mailto:jdeolive@anonymised.com>
@j_deolive <https://twitter.com/j_deolive&gt;

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer
CSIRO Earth Science and Resource Engineering
Australian Resources Research Centre

8

Just an update on this.
I think we could force all geotools and geoserver tests to create temporary files under their respective target/ directory at a very low cost with the following maven-surefire-plugin configuration in the root pom:

org.apache.maven.plugins maven-surefire-plugin 2.16 .... ${project.build.directory}

I’m testing with geotools and so far it looks like it works just fine.

···

On Fri, Mar 21, 2014 at 2:31 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

–

Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

On Fri, Mar 21, 2014 at 2:16 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I used it with an overridden create() method to create the temp folder under target/ before just for the same reason. It would probably be a good idea to use that trick in a geotools subclass. Nice thing about it is you don’t need to write anything in setUp/tearDown, just define the TemporaryFolder as an instance variable with the @Rule annotation. In any case, just saying, it could be done any other way too.

–

Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

On Fri, Mar 21, 2014 at 5:41 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

I’m pretty sure one of the reasons why it runs out of disk space is that both the geoserver and geotools builds leave a lot of dangling files and directories in /tmp. It usually has thousands of them. I’d be cool if we could do a review of the test cases in both projects to make sure temporary resources get deleted. A good way to do so is to use JUnit4’s TemporaryFolder utility: <https://weblogs.java.net/blog/johnsmart/archive/2009/09/29/working-temporary-files-junit-47>

Our current way of doing things is to have temp files be created under /target, but not all tests do that unfortunately.

The reason is that it’s often helpful to be able to analyze the files after the test has run, something that the junit
support does not seem to allow

Justin, do you have a list of file names that are getting generated, hopefully that will make it easier to pinpoint
the modules that are generating them?

Cheers

Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

9

On Mon, Mar 24, 2014 at 10:32 PM, Gabriel Roldan
<groldan@anonymised.com>wrote:

Just an update on this.
I think we could force all geotools and geoserver tests to create
temporary files under their respective target/ directory at a very low cost
with the following maven-surefire-plugin configuration in the root pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.16</version>
  <configuration>
    ....
    <systemPropertyVariables>
      <java.io.tmpdir>${project.build.directory}</java.io.tmpdir>
    </systemPropertyVariables>
  </configuration>
</plugin>

I'm testing with geotools and so far it looks like it works just fine.

Interesting, thanks for this.
Was it ever adopted?

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

10

I didn’t act on it but I don’t see any problem with trying it out.

···

On Wed, Mar 26, 2014 at 6:03 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

On Mon, Mar 24, 2014 at 10:32 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

Just an update on this.
I think we could force all geotools and geoserver tests to create temporary files under their respective target/ directory at a very low cost with the following maven-surefire-plugin configuration in the root pom:

org.apache.maven.plugins maven-surefire-plugin 2.16 .... ${project.build.directory}

I’m testing with geotools and so far it looks like it works just fine.

Interesting, thanks for this.
Was it ever adopted?

Cheers

Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

11

On Wed, Mar 26, 2014 at 9:03 AM, Andrea Aime
<andrea.aime@anonymised.com> wrote:

On Mon, Mar 24, 2014 at 10:32 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

Just an update on this.
I think we could force all geotools and geoserver tests to create
temporary files under their respective target/ directory at a very low cost
with the following maven-surefire-plugin configuration in the root pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.16</version>
  <configuration>
    ....
    <systemPropertyVariables>
      <java.io.tmpdir>${project.build.directory}</java.io.tmpdir>
    </systemPropertyVariables>
  </configuration>
</plugin>

I'm testing with geotools and so far it looks like it works just fine.

Interesting, thanks for this.
Was it ever adopted?

I have adopted it in geogit and its working fine.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

12

On Wed, Mar 26, 2014 at 2:15 PM, Gabriel Roldan
<groldan@anonymised.com> wrote:

On Wed, Mar 26, 2014 at 9:03 AM, Andrea Aime
<andrea.aime@anonymised.com> wrote:

On Mon, Mar 24, 2014 at 10:32 PM, Gabriel Roldan <groldan@anonymised.com>
wrote:

Just an update on this.
I think we could force all geotools and geoserver tests to create
temporary files under their respective target/ directory at a very low cost
with the following maven-surefire-plugin configuration in the root pom:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.16</version>
  <configuration>
    ....
    <systemPropertyVariables>
      <java.io.tmpdir>${project.build.directory}</java.io.tmpdir>
    </systemPropertyVariables>
  </configuration>
</plugin>

I'm testing with geotools and so far it looks like it works just fine.

Interesting, thanks for this.
Was it ever adopted?

I have adopted it in geogit and its working fine.

Hit send too soon.

I think it's not going to be that straightforward for
geotools/geoserver though, cause there seems to be a lot of ad-hoc
test code. I know for instance that it didn't work out of the box with
geoserver due to some conflicts with assumptions in the geoserver
tests infrastructure, but I can't exactly recall what the issue was.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan

--
Gabriel Roldán

Software Developer | Boundless

groldan@anonymised.com

@gabrielroldan