[Geoserver-devel] Random WMS failure on build box

Hi,
every few builds we are getting this failure, it seems to be random:

[INFO] Compiling 28 source files to /var/lib/jenkins/workspace/geoserver-master/src/wcs2_0/target/test-classes
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.289 sec <<< FAILURE!
testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest)  Time elapsed: 25 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[image/png]> but was:<[application/vnd.ogc.se_xml]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1249)
	at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Anyone else getting it? I cannot reproduce it locally

Cheers
Andrea

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


I believe I have run into it a couple of times, but not reliably.

Torben

···

On Fri, Jan 13, 2017 at 3:55 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
every few builds we are getting this failure, it seems to be random:

[INFO] Compiling 28 source files to /var/lib/jenkins/workspace/geoserver-master/src/wcs2_0/target/test-classes
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.289 sec <<< FAILURE!
testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest)  Time elapsed: 25 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[image/png]> but was:<[application/vnd.ogc.se_xml]>
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1249)
	at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Anyone else getting it? I cannot reproduce it locally

Cheers
Andrea

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On 14/01/17 00:55, Andrea Aime wrote:

testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest)
Time elapsed: 25 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[image/png]> but
was:<[application/vnd.ogc.se_xml]>

Andrea, I have seen this failure locally only once (OpenJDK 8 on Debian unstable). Jenkins sees it much more often.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Ok, I’ll switch that build off quiet tests, hopefully a stack trace will show up then :slight_smile:

Cheers
Andrea

···

On Fri, Jan 13, 2017 at 7:03 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

On 14/01/17 00:55, Andrea Aime wrote:

testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest)
Time elapsed: 25 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[image/png]> but
was:<[application/vnd.ogc.se_xml]>

Andrea, I have seen this failure locally only once (OpenJDK 8 on Debian unstable). Jenkins sees it much more often.

Kind regards,


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Andrea,

I can now reproduce this failure locally 100% of the time (10 out of 10 trials, running all gs-wms tests in Maven). The OWS ServiceExceptionReport (see below for details) contains:

org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation

The failure can be fixed locally (10 out of 10 trials, running all gs-wms tests in Maven) by reverting d15b9d8410ea636f76b8909dd5d3a0cae4bc138b on GeoTools master and recompiling gt-metadata:

[GEOT-5602] Better concurrent SoftValueHashMap
https://github.com/geotools/geotools/commit/d15b9d8410ea636f76b8909dd5d3a0cae4bc138b

See:
https://github.com/geotools/geotools/pull/1424
https://osgeo-org.atlassian.net/browse/GEOT-5602

We started seeing these failures soon after this PR was merged.

I obtained the OWS ServiceExceptionReport with this local change:

diff --git a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
index e7035feb4..eb97add5a 100644
--- a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
+++ b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
@@ -1246,7 +1246,7 @@ public class GeoServerSystemTestSupport extends GeoServerBaseTestSupport<SystemT
       */
      protected BufferedImage getAsImage(String path, String mime) throws Exception {
          MockHttpServletResponse resp = getAsServletResponse(path);
- assertEquals(mime, resp.getContentType());
+ assertEquals(resp.getContentAsString(), mime, resp.getContentType());
          InputStream is = getBinaryInputStream(resp);
          return ImageIO.read(is);
      }

I then saw this:

Running org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.718 sec <<< FAILURE!
testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest) Time elapsed: 15 sec <<< FAILURE!
org.junit.ComparisonFailure: <?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE ServiceExceptionReport SYSTEM "http://localhost:8080/geoserver/schemas/wms/1.1.1/WMS_exception_1_1_1.dtd&quot;&gt; <ServiceExceptionReport version="1.1.1" > <ServiceException code="internalError">
       Rendering process failed
org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation
</ServiceException></ServiceExceptionReport> expected:<[image/png]> but was:<[application/vnd.ogc.se_xml]>
  at org.junit.Assert.assertEquals(Assert.java:115)
  at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1249)
  at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hi Ben,
thanks a lot for the analysis! Any chance you can share a stack trace?
For the life of me I cannot get it to fail on my computer, so a full stack trace is key to solve
what seems to be a simple issue.

I would like to revert the pull request to allow Ugo quiet time during working hours to look at it, but I’m hesitant to do so without
a firm way to reproduce and thus analyze (cannot here) or at least a stack trace that would point us at the problem.

Cheers
Andrea

···

On Fri, Jan 13, 2017 at 11:56 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Andrea,

I can now reproduce this failure locally 100% of the time (10 out of 10 trials, running all gs-wms tests in Maven). The OWS ServiceExceptionReport (see below for details) contains:

org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation

The failure can be fixed locally (10 out of 10 trials, running all gs-wms tests in Maven) by reverting d15b9d8410ea636f76b8909dd5d3a0cae4bc138b on GeoTools master and recompiling gt-metadata:

[GEOT-5602] Better concurrent SoftValueHashMap
https://github.com/geotools/geotools/commit/d15b9d8410ea636f76b8909dd5d3a0cae4bc138b

See:
https://github.com/geotools/geotools/pull/1424
https://osgeo-org.atlassian.net/browse/GEOT-5602

We started seeing these failures soon after this PR was merged.

I obtained the OWS ServiceExceptionReport with this local change:

diff --git a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
index e7035feb4…eb97add5a 100644
— a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
+++ b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
@@ -1246,7 +1246,7 @@ public class GeoServerSystemTestSupport extends GeoServerBaseTestSupport<SystemT
*/
protected BufferedImage getAsImage(String path, String mime) throws Exception {
MockHttpServletResponse resp = getAsServletResponse(path);

  • assertEquals(mime, resp.getContentType());
  • assertEquals(resp.getContentAsString(), mime, resp.getContentType());
    InputStream is = getBinaryInputStream(resp);
    return ImageIO.read(is);
    }

I then saw this:

Running org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.718 sec <<< FAILURE!
testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest) Time elapsed: 15 sec <<< FAILURE!
org.junit.ComparisonFailure: <?xml version="1.0" encoding="UTF-8" standalone="no"?>
Rendering process failed
org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation
expected:<[image/png]> but was:<[application/vnd.ogc.se_xml]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1249)
at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)

Kind regards,


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Aaah, finally got it:

14 Jan 11:17:43 ERROR [geotools.rendering] - org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation
java.lang.ClassCastException: org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation
	at org.geotools.referencing.operation.BufferedCoordinateOperationFactory.createOperation(BufferedCoordinateOperationFactory.java:256)
	at org.geotools.geometry.jts.ReferencedEnvelope.transform(ReferencedEnvelope.java:688)
	at org.geotools.geometry.jts.ReferencedEnvelope.transform(ReferencedEnvelope.java:635)
	at org.geotools.renderer.crs.ProjectionHandler.preProcess(ProjectionHandler.java:417)
	at org.geotools.renderer.lite.StreamingRenderer$RenderableFeature.getTransformedShape(StreamingRenderer.java:3117)
	at org.geotools.renderer.lite.StreamingRenderer$RenderableFeature.getShape(StreamingRenderer.java:3077)
	at org.geotools.renderer.lite.StreamingRenderer.processSymbolizers(StreamingRenderer.java:2549)
	at org.geotools.renderer.lite.StreamingRenderer.processFeature(StreamingRenderer.java:2449)
	at org.geotools.renderer.lite.StreamingRenderer.drawPlain(StreamingRenderer.java:2306)
	at org.geotools.renderer.lite.StreamingRenderer.processStylers(StreamingRenderer.java:1927)
	at org.geotools.renderer.lite.StreamingRenderer.paint(StreamingRenderer.java:831)
	at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:545)
	at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:268)
	at org.geoserver.wms.map.RenderedImageMapOutputFormat.produceMap(RenderedImageMapOutputFormat.java:138)
	at org.geoserver.wms.GetMap.executeInternal(GetMap.java:623)
	at org.geoserver.wms.GetMap.run(GetMap.java:279)
	at org.geoserver.wms.GetMap.run(GetMap.java:125)
	at org.geoserver.wms.DefaultWebMapService.getMap(DefaultWebMapService.java:320)
	at sun.reflect.GeneratedMethodAccessor682.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.geoserver.ows.util.RequestObjectLogger.invoke(RequestObjectLogger.java:33)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	at com.sun.proxy.$Proxy44.getMap(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor683.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.geoserver.ows.Dispatcher.execute(Dispatcher.java:850)
	at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:268)
	at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.geoserver.test.GeoServerSystemTestSupport$1.service(GeoServerSystemTestSupport.java:1478)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.springframework.mock.web.MockFilterChain$ServletFilterProxy.doFilter(MockFilterChain.java:167)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.geoserver.test.GeoServerSystemTestSupport.dispatch(GeoServerSystemTestSupport.java:1505)
	at org.geoserver.test.GeoServerSystemTestSupport.dispatch(GeoServerSystemTestSupport.java:1436)
	at org.geoserver.test.GeoServerSystemTestSupport.getAsServletResponse(GeoServerSystemTestSupport.java:977)
	at org.geoserver.test.GeoServerSystemTestSupport.getAsServletResponse(GeoServerSystemTestSupport.java:960)
	at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1248)
	at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)

Cheers
Andrea
···

On Sat, Jan 14, 2017 at 11:31 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi Ben,
thanks a lot for the analysis! Any chance you can share a stack trace?
For the life of me I cannot get it to fail on my computer, so a full stack trace is key to solve
what seems to be a simple issue.

I would like to revert the pull request to allow Ugo quiet time during working hours to look at it, but I’m hesitant to do so without
a firm way to reproduce and thus analyze (cannot here) or at least a stack trace that would point us at the problem.

Cheers

Andrea

On Fri, Jan 13, 2017 at 11:56 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Andrea,

I can now reproduce this failure locally 100% of the time (10 out of 10 trials, running all gs-wms tests in Maven). The OWS ServiceExceptionReport (see below for details) contains:

org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation

The failure can be fixed locally (10 out of 10 trials, running all gs-wms tests in Maven) by reverting d15b9d8410ea636f76b8909dd5d3a0cae4bc138b on GeoTools master and recompiling gt-metadata:

[GEOT-5602] Better concurrent SoftValueHashMap
https://github.com/geotools/geotools/commit/d15b9d8410ea636f76b8909dd5d3a0cae4bc138b

See:
https://github.com/geotools/geotools/pull/1424
https://osgeo-org.atlassian.net/browse/GEOT-5602

We started seeing these failures soon after this PR was merged.

I obtained the OWS ServiceExceptionReport with this local change:

diff --git a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
index e7035feb4…eb97add5a 100644
— a/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
+++ b/src/main/src/test/java/org/geoserver/test/GeoServerSystemTestSupport.java
@@ -1246,7 +1246,7 @@ public class GeoServerSystemTestSupport extends GeoServerBaseTestSupport<SystemT
*/
protected BufferedImage getAsImage(String path, String mime) throws Exception {
MockHttpServletResponse resp = getAsServletResponse(path);

  • assertEquals(mime, resp.getContentType());
  • assertEquals(resp.getContentAsString(), mime, resp.getContentType());
    InputStream is = getBinaryInputStream(resp);
    return ImageIO.read(is);
    }

I then saw this:

Running org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest
Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.718 sec <<< FAILURE!
testAdvancedProjectionHandling(org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest) Time elapsed: 15 sec <<< FAILURE!
org.junit.ComparisonFailure: <?xml version="1.0" encoding="UTF-8" standalone="no"?>
Rendering process failed
org.geotools.util.SoftValueHashMap$Reference cannot be cast to org.opengis.referencing.operation.CoordinateOperation
expected:<[image/png]> but was:<[application/vnd.ogc.se_xml]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.geoserver.test.GeoServerSystemTestSupport.getAsImage(GeoServerSystemTestSupport.java:1249)
at org.geoserver.wms.wms_1_1_1.GetMapIntegrationTest.testAdvancedProjectionHandling(GetMapIntegrationTest.java:1094)

Kind regards,


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.