[Geoserver-devel] Intermittent file deletion errors on ares Jenkins

GeoServer GeoServer Developer
1

We often get failures with these:

Tests run: 25, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.457 sec <<< FAILURE!
org.geoserver.wfs.v2_0.GetFeatureJoinTest Time elapsed: 56 sec <<< ERROR!
java.io.IOException: Unable to delete directory ./target/default6619173959064498977data.

Any idea what might be causing them?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

2

Hi Ben,
I have been wondering myself as well… normally *nix systems have no notion of locking
so a deletion on a local file system should not be disallowed.

However, I’ve seen it happen on NFS mounts for files that were being vi edited from
another terminal. That’s clearly not the case here, but wondering if maybe
there is a remote system mount and maybe there is a thread in the process
that failed to release a file somehow, or another process maybe scanning those
directories while the build runs?

Wondering if setting “testdata.force.delete=true” might help here?
https://github.com/geoserver/geoserver/blob/master/src/main/src/test/java/org/geoserver/data/test/SystemTestData.java#L1007

(assuming the failure to delete comes from there, that is…)

Cheers
Andrea

···

On Thu, Apr 27, 2017 at 6:18 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

We often get failures with these:

Tests run: 25, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.457
sec <<< FAILURE!
org.geoserver.wfs.v2_0.GetFeatureJoinTest Time elapsed: 56 sec <<< ERROR!
java.io.IOException: Unable to delete directory
./target/default6619173959064498977data.

Any idea what might be causing them?

Kind regards,

–
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

I just saw this locally in Eclipse while trying to run a GeoServer test method. No NFS, nothing else accessing this:

java.io.IOException: Unable to delete directory ./target/default6787978788767592945data.
  at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1337)
  at org.geoserver.data.test.SystemTestData.tearDown(SystemTestData.java:1026)
  at org.geoserver.test.GeoServerBaseTestSupport.doTearDownClass(GeoServerBaseTestSupport.java:206)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
  at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
  at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
  at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

On 27/04/17 19:42, Andrea Aime wrote:

Hi Ben,
I have been wondering myself as well... normally *nix systems have no
notion of locking
so a deletion on a local file system should not be disallowed.

However, I've seen it happen on NFS mounts for files that were being vi
edited from
another terminal. That's clearly not the case here, but wondering if maybe
there is a remote system mount and _maybe_ there is a thread in the process
that failed to release a file somehow, or another process maybe scanning
those
directories while the build runs?

Wondering if setting "testdata.force.delete=true" might help here?
https://github.com/geoserver/geoserver/blob/master/src/main/src/test/java/org/geoserver/data/test/SystemTestData.java#L1007

(assuming the failure to delete comes from there, that is...)

Cheers
Andrea

On Thu, Apr 27, 2017 at 6:18 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

We often get failures with these:

Tests run: 25, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.457
sec <<< FAILURE!
org.geoserver.wfs.v2_0.GetFeatureJoinTest Time elapsed: 56 sec <<<
ERROR!
java.io.IOException: Unable to delete directory
./target/default6619173959064498977data.

Any idea what might be causing them?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

4

On Fri, Apr 28, 2017 at 5:37 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I just saw this locally in Eclipse while trying to run a GeoServer test
method. No NFS, nothing else accessing this:

java.io.IOException: Unable to delete directory
./target/default6787978788767592945data.
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.
java:1337)

On a local system hum... what can it be? Maybe the directory is gone
already?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

On 28/04/17 19:24, Andrea Aime wrote:

On Fri, Apr 28, 2017 at 5:37 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I just saw this locally in Eclipse while trying to run a GeoServer test
method. No NFS, nothing else accessing this:
java.io.IOException: Unable to delete directory
./target/default6787978788767592945data.
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.
java:1337)

On a local system hum... what can it be? Maybe the directory is gone
already?

We could take the easy way out and change tearDown to use deleteQuietly but it would be nice to know the cause, especially after so much effort was invested in getting it to work on Windows.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

6

On 29/04/17 13:12, Ben Caradoc-Davies wrote:

On 28/04/17 19:24, Andrea Aime wrote:

On Fri, Apr 28, 2017 at 5:37 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I just saw this locally in Eclipse while trying to run a GeoServer test
method. No NFS, nothing else accessing this:
java.io.IOException: Unable to delete directory
./target/default6787978788767592945data.
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.
java:1337)

On a local system hum... what can it be? Maybe the directory is gone
already?

We could take the easy way out and change tearDown to use deleteQuietly
but it would be nice to know the cause, especially after so much effort
was invested in getting it to work on Windows.

I think I have found the cause. GeoServerImpl is never disposed::

[GEOS-8122] Intermittent failure of FileUtils.deleteDirectory in SystemTestData.tearDown
https://osgeo-org.atlassian.net/browse/GEOS-8122

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

7

On 03/05/17 13:36, Ben Caradoc-Davies wrote:

I think I have found the cause. GeoServerImpl is never disposed::

[GEOS-8122] Intermittent failure of FileUtils.deleteDirectory in
SystemTestData.tearDown
https://osgeo-org.atlassian.net/browse/GEOS-8122

Pull request:
https://github.com/geoserver/geoserver/pull/2301

My next question is, was this defect the real cause of all those failures on Windows? Now that data stores are properly disposed, is the special handling for Windows in SystemTestData.tearDown() obsolete?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

8

Hi Ben,
that’s awesome… I just need to ask, how did you get to the conclusion that those
two were the causes? :slight_smile: Did you managed somehow to reproduce locally?

Cheers
Andrea

···

On Wed, May 3, 2017 at 5:38 AM, Ben Caradoc-Davies <ben@anonymised.com191…> wrote:

On 03/05/17 13:36, Ben Caradoc-Davies wrote:

I think I have found the cause. GeoServerImpl is never disposed::

[GEOS-8122] Intermittent failure of FileUtils.deleteDirectory in
SystemTestData.tearDown
https://osgeo-org.atlassian.net/browse/GEOS-8122

Pull request:
https://github.com/geoserver/geoserver/pull/2301

My next question is, was this defect the real cause of all those failures on Windows? Now that data stores are properly disposed, is the special handling for Windows in SystemTestData.tearDown() obsolete?

Kind regards,

–
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

9

Ah,
for Windows I believe we still need the extra checks as coverage files can still be stuck with a thread keeping them open for whatever reason (faulty test, bug in GeoServer teardown). Generally speaking, as we don’t have an active core developer on Windows and these stuff tends to be race condition related (“works on mine, fails on yours” types of stuff) I would not remove the Windows specific behavior.

Cheers
Andrea

···

On Wed, May 3, 2017 at 11:52 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi Ben,
that’s awesome… I just need to ask, how did you get to the conclusion that those
two were the causes? :slight_smile: Did you managed somehow to reproduce locally?

Cheers

Andrea

–

On Wed, May 3, 2017 at 5:38 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

On 03/05/17 13:36, Ben Caradoc-Davies wrote:

I think I have found the cause. GeoServerImpl is never disposed::

[GEOS-8122] Intermittent failure of FileUtils.deleteDirectory in
SystemTestData.tearDown
https://osgeo-org.atlassian.net/browse/GEOS-8122

Pull request:
https://github.com/geoserver/geoserver/pull/2301

My next question is, was this defect the real cause of all those failures on Windows? Now that data stores are properly disposed, is the special handling for Windows in SystemTestData.tearDown() obsolete?

Kind regards,

–
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

10

Andrea,

I could not reproduce locally, but while trying to do so running org.geoserver.wfs.v2_0.GetFeatureJoinTest in Eclipse, I noticed that src/wfs/target/default7740998516718894422data/foo.2.log.db was being recreated after the data directory was deleted, which means that the H2 test database was not being disposed before the directory deletion. Although the application context is destroyed, breakpoints showed that CatalogImpl.dispose() and GeoServerImpl.dispose() are never called, so the H2 data store can go on writing logs. The only thing that will stop a directory from being deleted on Linux is if it is not empty, and this behaviour is a likely culprit. Adding a call to GeoServer.dispose() in GeoServerSystemTestSupport.destroyGeoServer() ensures that all data stores are disposed before the data directory is removed. This fix immediately stopped the writing of foo.2.log.db and data directories were fully cleaned up after GetFeatureJoinTest.

After applying the GeoServerSystemTestSupport.destroyGeoServer() fix, GWCIntegrationTest started failing deterministically in both Maven and Eclipse. Individual methods pass, but run as a suite, many failed, which is clear evidence of a leaky fixture. This test has TestSetupFrequency.REPEAT so we expect fresh beans in our Spring application context. The GWC bean (static singleton) was being reconstructed correctly, but the static tile encoder cache was hanging onto encoders: every test used encoders from whichever test class was first to use GWC.getResponseEncoder, and these could even be other test suites in the module as the static cache will persist once the class has been loaded. The stale encoders were revealed by the GeoServerSystemTestSupport.destroyGeoServer() fix, which ensured that the stale beans were disposed, breaking the leaky test. The fix was to make GWC.cachedTileEncoders a member rather than static, to ensure that each new GWC singleton gets a fresh cache.

The fixes are tiny: GEOS-8122 is a one-line fix that adds 33 characters, and GEOS-8123 is a one-line fix that removes 7 characters. These are dwarfed by the write-up. :slight_smile:

Kind regards,
Ben.

On 03/05/17 21:52, Andrea Aime wrote:

Hi Ben,
that's awesome... I just need to ask, how did you get to the conclusion
that those
two were the causes? :slight_smile: Did you managed somehow to reproduce locally?

Cheers
Andrea

On Wed, May 3, 2017 at 5:38 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

On 03/05/17 13:36, Ben Caradoc-Davies wrote:

I think I have found the cause. GeoServerImpl is never disposed::

[GEOS-8122] Intermittent failure of FileUtils.deleteDirectory in
SystemTestData.tearDown
https://osgeo-org.atlassian.net/browse/GEOS-8122

Pull request:
https://github.com/geoserver/geoserver/pull/2301

My next question is, was this defect the real cause of all those failures
on Windows? Now that data stores are properly disposed, is the special
handling for Windows in SystemTestData.tearDown() obsolete?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand