[Geoserver-users] GeoServer resource over-use on GetMap raster warping

[Jonathan_Moules2]

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and pyramided) geotiffs which make up an ImageMosaic. The imagery is in EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is incapable of automatically using the native projection and instead requests the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated RAM, uses 100% of one core of the CPU it’s on, and becomes generally much less responsive. The request never seems to time out, it just keeps processing indefinitely.

I appreciate that warping is a computationally hard problem, but shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.

The tomcat localhost_access.log doesn’t even show the request, even 11 mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and it’s impossible to get ArcGIS to behave properly, so is there another solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[ianturton]

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

···

On 27 June 2014 13:19, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and pyramided) geotiffs which make up an ImageMosaic. The imagery is in EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is incapable of automatically using the native projection and instead requests the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated RAM, uses 100% of one core of the CPU it’s on, and becomes generally much less responsive. The request never seems to time out, it just keeps processing indefinitely.

I appreciate that warping is a computationally hard problem, but shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.

The tomcat localhost_access.log doesn’t even show the request, even 11 mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and it’s impossible to get ArcGIS to behave properly, so is there another solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

[Jonathan_Moules2]

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also support geographic coordinates by geocentric coordinate systems such as “CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based systems.

ESRI took this to mean “we should always use this EPSG by default”, hence I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and pyramided) geotiffs which make up an ImageMosaic. The imagery is in EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is incapable of automatically using the native projection and instead requests the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated RAM, uses 100% of one core of the CPU it’s on, and becomes generally much less responsive. The request never seems to time out, it just keeps processing indefinitely.

I appreciate that warping is a computationally hard problem, but shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.

The tomcat localhost_access.log doesn’t even show the request, even 11 mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and it’s impossible to get ArcGIS to behave properly, so is there another solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

[ianturton]

That is clearly a should not a must (or shall) - so it’s optional. File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

···

On 27 June 2014 15:00, Jonathan Moules <jonathanmoules@anonymised.com4942…> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

–
Ian Turton

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also support geographic coordinates by geocentric coordinate systems such as “CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based systems.

ESRI took this to mean “we should always use this EPSG by default”, hence I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and pyramided) geotiffs which make up an ImageMosaic. The imagery is in EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is incapable of automatically using the native projection and instead requests the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated RAM, uses 100% of one core of the CPU it’s on, and becomes generally much less responsive. The request never seems to time out, it just keeps processing indefinitely.

I appreciate that warping is a computationally hard problem, but shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.

The tomcat localhost_access.log doesn’t even show the request, even 11 mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and it’s impossible to get ArcGIS to behave properly, so is there another solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

[Jonathan_Moules2]

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

Hehe, have you ever tried filing a bug report with ESRI? I’ll pass on that one.

On the whole we’re happy to have GeoServer functioning with WGS:84, that’s what the KML users required. However for this particular dataset it’s causing issues.
I guess there’s no way to restrict projections per layer?

Storing the layer twice is suboptimal. It’s by far our biggest layer at 30GB of server space.

Is this expected behaviour when warping a raster? Shouldn’t it time out? I understand WCS persevering, but not so much WMS.

Cheers,
Jonathan

On 27 June 2014 15:06, Ian Turton <ijturton@anonymised.com> wrote:

That is clearly a should not a must (or shall) - so it’s optional. File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

On 27 June 2014 15:00, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

–

Ian Turton

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also support geographic coordinates by geocentric coordinate systems such as “CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based systems.

ESRI took this to mean “we should always use this EPSG by default”, hence I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and pyramided) geotiffs which make up an ImageMosaic. The imagery is in EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is incapable of automatically using the native projection and instead requests the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated RAM, uses 100% of one core of the CPU it’s on, and becomes generally much less responsive. The request never seems to time out, it just keeps processing indefinitely.

I appreciate that warping is a computationally hard problem, but shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.

The tomcat localhost_access.log doesn’t even show the request, even 11 mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and it’s impossible to get ArcGIS to behave properly, so is there another solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

[Simone_Giannecchini3]

Chiming in and actually interrupting the thread, apologies

What you report should not happen regardless of the reprojection.

I would need to know:

- version fo geoseerver
- wms resource limits configuration (especially max rendering time)
- mosaic configuration ( I mean the configuration parameter for the msoaic)
- structure of the underlying data

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

On Fri, Jun 27, 2014 at 4:29 PM, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hehe, have you ever tried filing a bug report with ESRI? I'll pass on that
one.

On the whole we're happy to have GeoServer functioning with WGS:84, that's
what the KML users required. However for this particular dataset it's
causing issues.
I guess there's no way to restrict projections per layer?

Storing the layer twice is suboptimal. It's by far our biggest layer at 30GB
of server space.

Is this expected behaviour when warping a raster? Shouldn't it time out? I
understand WCS persevering, but not so much WMS.

Cheers,
Jonathan

On 27 June 2014 15:06, Ian Turton <ijturton@anonymised.com> wrote:

That is clearly a *should* not a *must* (or *shall*) - so it's optional.
File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

On 27 June 2014 15:00, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi Ian,
Yep, it's 6.7.3.1:

To maximize interoperability among servers, providers should also
support geographic coordinates by geocentric coordinate systems such as
“CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based
systems.

ESRI took this to mean "we should always use this EPSG by default", hence
I end up with this mess. It doesn't seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about "it's in the WMS spec as mandatory" ? I'm fairly sure
it isn't though I don't use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and
pyramided) geotiffs which make up an ImageMosaic. The imagery is in
EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is
incapable of automatically using the native projection and instead requests
the layer in EPSG:4326 whenever it's added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3AAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated
RAM, uses 100% of one core of the CPU it's on, and becomes generally much
less responsive. The request never seems to time out, it just keeps
processing indefinitely.

I appreciate that warping is a computationally hard problem, but
shouldn't there be some sort of built in clean up to stop this?
My WMS "resource consumption limits" are all at the defaults.
The tomcat localhost_access.log doesn't even show the request, even 11
mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It's impossible to
remove EPSG:4326 from the CRS list (it's in the WMS spec as mandatory), and
it's impossible to get ArcGIS to behave properly, so is there another
solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community
Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

--
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Jonathan_Moules2]

Hi Simone,
That’s “good” to hear (in a certain context ).

To answer your questions:

• GeoServer 2.5.1

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60
Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 27 June 2014 15:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Chiming in and actually interrupting the thread, apologies

What you report should not happen regardless of the reprojection.

I would need to know:

• version fo geoseerver
• wms resource limits configuration (especially max rendering time)
• mosaic configuration ( I mean the configuration parameter for the msoaic)
• structure of the underlying data

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Fri, Jun 27, 2014 at 4:29 PM, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hehe, have you ever tried filing a bug report with ESRI? I’ll pass on that
one.

On the whole we’re happy to have GeoServer functioning with WGS:84, that’s
what the KML users required. However for this particular dataset it’s
causing issues.
I guess there’s no way to restrict projections per layer?

Storing the layer twice is suboptimal. It’s by far our biggest layer at 30GB
of server space.

Is this expected behaviour when warping a raster? Shouldn’t it time out? I
understand WCS persevering, but not so much WMS.

Cheers,
Jonathan

On 27 June 2014 15:06, Ian Turton <ijturton@anonymised.com> wrote:

That is clearly a should not a must (or shall) - so it’s optional.
File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

On 27 June 2014 15:00, Jonathan Moules
<jonathanmoules@anonymised.com.4942…> wrote:

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also
support geographic coordinates by geocentric coordinate systems such as
“CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based
systems.

ESRI took this to mean “we should always use this EPSG by default”, hence
I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure
it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and
pyramided) geotiffs which make up an ImageMosaic. The imagery is in
EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is
incapable of automatically using the native projection and instead requests
the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated
RAM, uses 100% of one core of the CPU it’s on, and becomes generally much
less responsive. The request never seems to time out, it just keeps
processing indefinitely.

I appreciate that warping is a computationally hard problem, but
shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.
The tomcat localhost_access.log doesn’t even show the request, even 11
mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to
remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and
it’s impossible to get ArcGIS to behave properly, so is there another
solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community
Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Simone_Giannecchini3]

Ciao Jonathan,
please, read below…

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Fri, Jun 27, 2014 at 5:03 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,
That’s “good” to hear (in a certain context ).

To answer your questions:

• GeoServer 2.5.1

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

On 27 June 2014 15:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Chiming in and actually interrupting the thread, apologies

What you report should not happen regardless of the reprojection.

I would need to know:

• version fo geoseerver
• wms resource limits configuration (especially max rendering time)
• mosaic configuration ( I mean the configuration parameter for the msoaic)
• structure of the underlying data

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Fri, Jun 27, 2014 at 4:29 PM, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hehe, have you ever tried filing a bug report with ESRI? I’ll pass on that
one.

On the whole we’re happy to have GeoServer functioning with WGS:84, that’s
what the KML users required. However for this particular dataset it’s
causing issues.
I guess there’s no way to restrict projections per layer?

Storing the layer twice is suboptimal. It’s by far our biggest layer at 30GB
of server space.

Is this expected behaviour when warping a raster? Shouldn’t it time out? I
understand WCS persevering, but not so much WMS.

Cheers,
Jonathan

On 27 June 2014 15:06, Ian Turton <ijturton@anonymised.com> wrote:

That is clearly a should not a must (or shall) - so it’s optional.
File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

On 27 June 2014 15:00, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also
support geographic coordinates by geocentric coordinate systems such as
“CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based
systems.

ESRI took this to mean “we should always use this EPSG by default”, hence
I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure
it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and
pyramided) geotiffs which make up an ImageMosaic. The imagery is in
EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is
incapable of automatically using the native projection and instead requests
the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated
RAM, uses 100% of one core of the CPU it’s on, and becomes generally much
less responsive. The request never seems to time out, it just keeps
processing indefinitely.

I appreciate that warping is a computationally hard problem, but
shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.
The tomcat localhost_access.log doesn’t even show the request, even 11
mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to
remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and
it’s impossible to get ArcGIS to behave properly, so is there another
solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community
Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Jonathan_Moules2]

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

Hi Simone,
Responses inline

It got to 20 mins before I killed it by restarting GeoServer. Standard “net stop geoserver_instance” didn’t work. Had to be forceful.

I can do better:
http:\maps.warwickshire.gov.uk\shr\AP_2012_2013_imageMosaic.zip - that file has two of the tif files in it. It’s about 100MB.

I’ll try and get onto that on Monday.

This is a Windows 2008 R2 server.

Cheers,
Jonathan

and the runaway request lasts for more than 60s?

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

On 27 June 2014 15:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Chiming in and actually interrupting the thread, apologies

What you report should not happen regardless of the reprojection.

I would need to know:

• version fo geoseerver
• wms resource limits configuration (especially max rendering time)
• mosaic configuration ( I mean the configuration parameter for the msoaic)
• structure of the underlying data

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Fri, Jun 27, 2014 at 4:29 PM, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hehe, have you ever tried filing a bug report with ESRI? I’ll pass on that
one.

On the whole we’re happy to have GeoServer functioning with WGS:84, that’s
what the KML users required. However for this particular dataset it’s
causing issues.
I guess there’s no way to restrict projections per layer?

Storing the layer twice is suboptimal. It’s by far our biggest layer at 30GB
of server space.

Is this expected behaviour when warping a raster? Shouldn’t it time out? I
understand WCS persevering, but not so much WMS.

Cheers,
Jonathan

On 27 June 2014 15:06, Ian Turton <ijturton@anonymised.com> wrote:

That is clearly a should not a must (or shall) - so it’s optional.
File a bug report against it.

Or a more practical note can you store the layer twice?

Ian

On 27 June 2014 15:00, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi Ian,
Yep, it’s 6.7.3.1:

To maximize interoperability among servers, providers should also
support geographic coordinates by geocentric coordinate systems such as
“CRS:84” (see 6.7.3.2), “EPSG:4326” (see 6.7.3.3) or other ITRF-based
systems.

ESRI took this to mean “we should always use this EPSG by default”, hence
I end up with this mess. It doesn’t seem to be in 1.1.0
Cheers,
Jonathan

On 27 June 2014 13:44, Ian Turton <ijturton@anonymised.com> wrote:

Are you sure about “it’s in the WMS spec as mandatory” ? I’m fairly sure
it isn’t though I don’t use 1.3 much so it may have slipped in there.

Ian

On 27 June 2014 13:19, Jonathan Moules
<jonathanmoules@anonymised.com> wrote:

Hi List,
My aerial imagery is 31GB, comprising 13 optimised (tiled and
pyramided) geotiffs which make up an ImageMosaic. The imagery is in
EPSG:27700 - British National Grid. This all works fine.

However, a certain piece of proprietary GIS software (namely ESRI) is
incapable of automatically using the native projection and instead requests
the layer in EPSG:4326 whenever it’s added.

The request is:

http://example.com/gs/ows?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&CRS=CRS:84&BBOX=-1.9868610903408341,51.938491047471814,-1.1430930819885086,52.705668101075581&WIDTH=919&HEIGHT=836&LAYERS=Aerial_Photography%3aAerial_Photography_2013&STYLES=&EXCEPTIONS=XML&FORMAT=image/png&BGCOLOR=0xFEFFFF&TRANSPARENT=TRUE

This then proceeds to hammer GeoServer which maxes out its allocated
RAM, uses 100% of one core of the CPU it’s on, and becomes generally much
less responsive. The request never seems to time out, it just keeps
processing indefinitely.

I appreciate that warping is a computationally hard problem, but
shouldn’t there be some sort of built in clean up to stop this?
My WMS “resource consumption limits” are all at the defaults.
The tomcat localhost_access.log doesn’t even show the request, even 11
mins later - I have to use Fiddler.

Any suggestions for how to handle this situation? It’s impossible to
remove EPSG:4326 from the CRS list (it’s in the WMS spec as mandatory), and
it’s impossible to get ArcGIS to behave properly, so is there another
solution that entails getting GeoServer to work smarter?

Cheers,
Jonathan

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community
Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

–
Ian Turton

This transmission is intended for the named addressee(s) only and may
contain confidential, sensitive or personal information and should be
handled accordingly. Unless you are the named addressee (or authorised to
receive it for the addressee) you may not copy or use it, or disclose it to
anyone else. If you have received this transmission in error please notify
the sender immediately. All email traffic sent to or from us, including
without limitation all GCSX traffic, may be subject to recording and/or
monitoring in accordance with relevant legislation.

---

Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

---

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Andrea_Aime4]

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <
simone.giannecchini@anonymised.com> wrote:

- WMS resource limits are the defaults. So:
Memory: 64000
Time: 60

and the runaway request lasts for more than 60s?

As far as I remember, this is completely normal unfortunately, we have no
way to stop raster processing chains, only
vector rendering ones: the limits are applied inside
RenderedImageMapOutputFormat, during
the "rendering" phase, but raster chains start actually processing when the
output
is encoded, so at the very last step, where we have no control over what's
going on.

To control this we'd need either a custom tile scheduler per request, with
a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts
throwing exceptions
when the encoder pulls tiles after the timeout... given than most output
formats just do a getData()
there is also the issue that we have no control of what the writer does
after it has got all
the tiles, and actually starts encoding the output... maybe we could have a
output stream
wrapper that also has a timeout...

Errors: 1000

- ImageMosaic parameters in screenshot:
!image.png|1243x584

- Structure of the underlying data. What do you mean? It's 13 irregularly
sized/shaped GeoTIFF images created by GDAL in the standard way. They each
contain tiles and overviews. The data is RGB Aerial photography compressed
with JPEG YCBR.

Hope that's enough information to be going on. I can provide more if
desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The
others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I'd like to see what threads are being blocked onto if we are not
respecting the 60s limit

I'm wondering if the overviews are being used given that the mosaic seems
to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for
this particular case.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Simone_Giannecchini3]

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Jonathan_Moules2]

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Ciao Jonathan,
one last info, how did you configure the CRS for this layer?

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Jonathan_Moules2]

Hi Simone,

• The source data is in 27700.
• However for whatever reason GeoServer notes the native SRS as “UNKNOWN”. The definition GeoServer shows is:

PROJCS[“unnamed”,
GEOGCS[“GCS Name = GCS_OSGB_1936”,
DATUM[“OSGB 1936”,
SPHEROID[“Airy 1830”, 6377563.396, 299.3249646, AUTHORITY[“EPSG”,“7001”]],
TOWGS84[446.448, -125.157, 542.06, 0.15, 0.247, 0.842, -20.489],
AUTHORITY[“EPSG”,“6277”]],
PRIMEM[“Greenwich”, 0.0, AUTHORITY[“EPSG”,“8901”]],
UNIT[“degree”, 0.017453292519943295],
AXIS[“Geodetic longitude”, EAST],
AXIS[“Geodetic latitude”, NORTH]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“central_meridian”, -2.0],
PARAMETER[“latitude_of_origin”, 49.0],
PARAMETER[“scale_factor”, 0.9996012717],
PARAMETER[“false_easting”, 400000.0],
PARAMETER[“false_northing”, -100000.0],
UNIT[“m”, 1.0],
AXIS[“Easting”, EAST],
AXIS[“Northing”, NORTH]]

• Declared is 27700, and the SRS handling is “force declared”.

Cheers,
Jonathan

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Ciao Jonathan,
there is not EPSG code in the definition of your tiff files hence GeoServer can not know that this is 27700 (it would be very expensive to make such a check).

In this situation I usually recommend putting a prj with the right CRS right next to each file which tell geotools/geoserver to use the external PRJ rather than the internal definition.
This is good because in some cases the different tif files have slighly different CRS hence you are risking that the internal harvester skip some of them.

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 2:37 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,

• The source data is in 27700.
• However for whatever reason GeoServer notes the native SRS as “UNKNOWN”. The definition GeoServer shows is:

PROJCS[“unnamed”,
GEOGCS[“GCS Name = GCS_OSGB_1936”,
DATUM[“OSGB 1936”,
SPHEROID[“Airy 1830”, 6377563.396, 299.3249646, AUTHORITY[“EPSG”,“7001”]],
TOWGS84[446.448, -125.157, 542.06, 0.15, 0.247, 0.842, -20.489],
AUTHORITY[“EPSG”,“6277”]],
PRIMEM[“Greenwich”, 0.0, AUTHORITY[“EPSG”,“8901”]],
UNIT[“degree”, 0.017453292519943295],
AXIS[“Geodetic longitude”, EAST],
AXIS[“Geodetic latitude”, NORTH]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“central_meridian”, -2.0],
PARAMETER[“latitude_of_origin”, 49.0],
PARAMETER[“scale_factor”, 0.9996012717],
PARAMETER[“false_easting”, 400000.0],
PARAMETER[“false_northing”, -100000.0],
UNIT[“m”, 1.0],
AXIS[“Easting”, EAST],
AXIS[“Northing”, NORTH]]

• Declared is 27700, and the SRS handling is “force declared”.

Cheers,
Jonathan

On 1 July 2014 17:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last info, how did you configure the CRS for this layer?

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Ciao Jonathan,
the jira is on GeoTools:

https://jira.codehaus.org/browse/GEOT-4836

A fix has been provided on 11.x and master which means 2.5.x and .master of GeoServer.

You should be able to test next nightly to pick up the change.

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 3:10 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
there is not EPSG code in the definition of your tiff files hence GeoServer can not know that this is 27700 (it would be very expensive to make such a check).

In this situation I usually recommend putting a prj with the right CRS right next to each file which tell geotools/geoserver to use the external PRJ rather than the internal definition.
This is good because in some cases the different tif files have slighly different CRS hence you are risking that the internal harvester skip some of them.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 2:37 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,

• The source data is in 27700.
• However for whatever reason GeoServer notes the native SRS as “UNKNOWN”. The definition GeoServer shows is:

PROJCS[“unnamed”,
GEOGCS[“GCS Name = GCS_OSGB_1936”,
DATUM[“OSGB 1936”,
SPHEROID[“Airy 1830”, 6377563.396, 299.3249646, AUTHORITY[“EPSG”,“7001”]],
TOWGS84[446.448, -125.157, 542.06, 0.15, 0.247, 0.842, -20.489],
AUTHORITY[“EPSG”,“6277”]],
PRIMEM[“Greenwich”, 0.0, AUTHORITY[“EPSG”,“8901”]],
UNIT[“degree”, 0.017453292519943295],
AXIS[“Geodetic longitude”, EAST],
AXIS[“Geodetic latitude”, NORTH]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“central_meridian”, -2.0],
PARAMETER[“latitude_of_origin”, 49.0],
PARAMETER[“scale_factor”, 0.9996012717],
PARAMETER[“false_easting”, 400000.0],
PARAMETER[“false_northing”, -100000.0],
UNIT[“m”, 1.0],
AXIS[“Easting”, EAST],
AXIS[“Northing”, NORTH]]

• Declared is 27700, and the SRS handling is “force declared”.

Cheers,
Jonathan

On 1 July 2014 17:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last info, how did you configure the CRS for this layer?

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Jonathan_Moules2]

Hi Simone,
Thanks for this. I can confirm it seems to be fixed with the July 7th build of 2.5.x.

Good point about no coordinate definition in the GeoTIff. I don’t know why not, that’s standard practice to add. Will have to rectify.

Thanks again,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

···

On 2 July 2014 19:52, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
the jira is on GeoTools:

https://jira.codehaus.org/browse/GEOT-4836

A fix has been provided on 11.x and master which means 2.5.x and .master of GeoServer.

You should be able to test next nightly to pick up the change.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 3:10 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
there is not EPSG code in the definition of your tiff files hence GeoServer can not know that this is 27700 (it would be very expensive to make such a check).

In this situation I usually recommend putting a prj with the right CRS right next to each file which tell geotools/geoserver to use the external PRJ rather than the internal definition.
This is good because in some cases the different tif files have slighly different CRS hence you are risking that the internal harvester skip some of them.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 2:37 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,

• The source data is in 27700.
• However for whatever reason GeoServer notes the native SRS as “UNKNOWN”. The definition GeoServer shows is:

PROJCS[“unnamed”,
GEOGCS[“GCS Name = GCS_OSGB_1936”,
DATUM[“OSGB 1936”,
SPHEROID[“Airy 1830”, 6377563.396, 299.3249646, AUTHORITY[“EPSG”,“7001”]],
TOWGS84[446.448, -125.157, 542.06, 0.15, 0.247, 0.842, -20.489],
AUTHORITY[“EPSG”,“6277”]],
PRIMEM[“Greenwich”, 0.0, AUTHORITY[“EPSG”,“8901”]],
UNIT[“degree”, 0.017453292519943295],
AXIS[“Geodetic longitude”, EAST],
AXIS[“Geodetic latitude”, NORTH]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“central_meridian”, -2.0],
PARAMETER[“latitude_of_origin”, 49.0],
PARAMETER[“scale_factor”, 0.9996012717],
PARAMETER[“false_easting”, 400000.0],
PARAMETER[“false_northing”, -100000.0],
UNIT[“m”, 1.0],
AXIS[“Easting”, EAST],
AXIS[“Northing”, NORTH]]

• Declared is 27700, and the SRS handling is “force declared”.

Cheers,
Jonathan

On 1 July 2014 17:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last info, how did you configure the CRS for this layer?

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit

[Simone_Giannecchini3]

Dear Jonathan,

thx for the feedback.

···

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 8, 2014 at 1:38 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,
Thanks for this. I can confirm it seems to be fixed with the July 7th build of 2.5.x.

Good point about no coordinate definition in the GeoTIff. I don’t know why not, that’s standard practice to add. Will have to rectify.

Thanks again,
Jonathan

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

On 2 July 2014 19:52, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
the jira is on GeoTools:

https://jira.codehaus.org/browse/GEOT-4836

A fix has been provided on 11.x and master which means 2.5.x and .master of GeoServer.

You should be able to test next nightly to pick up the change.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 3:10 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
there is not EPSG code in the definition of your tiff files hence GeoServer can not know that this is 27700 (it would be very expensive to make such a check).

In this situation I usually recommend putting a prj with the right CRS right next to each file which tell geotools/geoserver to use the external PRJ rather than the internal definition.
This is good because in some cases the different tif files have slighly different CRS hence you are risking that the internal harvester skip some of them.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Wed, Jul 2, 2014 at 2:37 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

Hi Simone,

• The source data is in 27700.
• However for whatever reason GeoServer notes the native SRS as “UNKNOWN”. The definition GeoServer shows is:

PROJCS[“unnamed”,
GEOGCS[“GCS Name = GCS_OSGB_1936”,
DATUM[“OSGB 1936”,
SPHEROID[“Airy 1830”, 6377563.396, 299.3249646, AUTHORITY[“EPSG”,“7001”]],
TOWGS84[446.448, -125.157, 542.06, 0.15, 0.247, 0.842, -20.489],
AUTHORITY[“EPSG”,“6277”]],
PRIMEM[“Greenwich”, 0.0, AUTHORITY[“EPSG”,“8901”]],
UNIT[“degree”, 0.017453292519943295],
AXIS[“Geodetic longitude”, EAST],
AXIS[“Geodetic latitude”, NORTH]],
PROJECTION[“Transverse_Mercator”],
PARAMETER[“central_meridian”, -2.0],
PARAMETER[“latitude_of_origin”, 49.0],
PARAMETER[“scale_factor”, 0.9996012717],
PARAMETER[“false_easting”, 400000.0],
PARAMETER[“false_northing”, -100000.0],
UNIT[“m”, 1.0],
AXIS[“Easting”, EAST],
AXIS[“Northing”, NORTH]]

• Declared is 27700, and the SRS handling is “force declared”.

Cheers,
Jonathan

On 1 July 2014 17:50, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last info, how did you configure the CRS for this layer?

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Tue, Jul 1, 2014 at 6:32 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao,
I confirm this is a bug.
We reproduced it locally thanks to the sample data and the request.
I would expect a fix shortly.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 3:18 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
please, don’t do anything more, we’ll replicate locally and then we’ll let you know.

Regards,
Simone Giannecchini

GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Mon, Jun 30, 2014 at 1:51 PM, Jonathan Moules <jonathanmoules@anonymised.com> wrote:

This transmission is intended for the named addressee(s) only and may contain confidential, sensitive or personal information and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

Hi Both,
Good point on the pyramid use; I was wondering why it was so slow given they exist. There was an issue a bit over a year ago that you fixed Simone where they simply weren’t being used at all if they were hetergenous.

The properties file contains:

#-Automagically created from GeoTools-
#Tue Nov 12 08:39:42 GMT 2013
Levels=0.125,0.125 0.25,0.25 0.5,0.5 1.0,1.0 2.0,2.0 4.0,4.0 8.0,8.0 16.0,15.873015873015873 31.914893617021278,31.25
Heterogeneous=true
AbsolutePath=false
Name=AP_2012_2013_imageMosaic
TypeName=AP_2012_2013_imageMosaic
Caching=false
ExpandToRGB=false
LocationAttribute=location
LevelsNum=9

Do you still me to try my hand at creating a dump?

Cheers,
Jonathan

On 28 June 2014 10:39, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

Ciao Jonathan,
one last thing to check as Andrea also pointed out is the mosaic.properties file that contains the config of the mosaic.

Could you please share it with us?

Regards,
Simone Giannecchini

==
GeoServer Professional Services from the experts! Visit http://goo.gl/NWWaa2 for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

On Sat, Jun 28, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Fri, Jun 27, 2014 at 5:43 PM, Simone Giannecchini <simone.giannecchini@anonymised.com> wrote:

As far as I remember, this is completely normal unfortunately, we have no way to stop raster processing chains, only
vector rendering ones: the limits are applied inside RenderedImageMapOutputFormat, during
the “rendering” phase, but raster chains start actually processing when the output
is encoded, so at the very last step, where we have no control over what’s going on.

To control this we’d need either a custom tile scheduler per request, with a timeout, or
some way to poison the chain, e.g. put a fake operation into it that starts throwing exceptions
when the encoder pulls tiles after the timeout… given than most output formats just do a getData()
there is also the issue that we have no control of what the writer does after it has got all
the tiles, and actually starts encoding the output… maybe we could have a output stream
wrapper that also has a timeout…

I’m wondering if the overviews are being used given that the mosaic seems to be heterogeneous.
I know there was a recent fix, but wondering if there are still issues for this particular case.

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

---

• WMS resource limits are the defaults. So:
  Memory: 64000

Time: 60

and the runaway request lasts for more than 60s?

Errors: 1000

• ImageMosaic parameters in screenshot:
  

• Structure of the underlying data. What do you mean? It’s 13 irregularly sized/shaped GeoTIFF images created by GDAL in the standard way. They each contain tiles and overviews. The data is RGB Aerial photography compressed with JPEG YCBR.

Hope that’s enough information to be going on. I can provide more if desired.

can you paste the gdalinfo on one of these files?

Moreover can you share also the JAI and Coverage settings?

That said, I would suggest replicate and take at least a jstack dump. The others quick dumps listed here would be of help:

http://docs.geoserver.org/latest/en/user/production/troubleshooting.html

I’d like to see what threads are being blocked onto if we are not respecting the 60s limit