[Geoserver-users] Performance issue with large polygon

GeoServer GeoServer User
1

Hi,

We have a polygon layer used as a backdrop for a WMS based layer group. The layer contains the outlines of the UK, one of which is the mainland and is obviously quite large. I have generalised versions of the geometries to use when zoomed out, however when zoomed in, encoding the whole mainland for every tile introduces a bottleneck and requests quickly build up and time out as every request. The layer group contains around 20 different layers, without the polygon layer the group renders very quickly add the layer back in and it becomes very unsresposive.

The layer is used to provide the coast and a background colour for land. I'm looking at "chopping" up the polygons on a grid to create smaller geometries that can be used to provide the background colour and extract the coast into a number of smaller lines.

But before I do this I was wondering if there was some way I could get geoserver to cache the features for this layer in memory rather than having to hit the database for every request.

Regards, Vic
--

________________________________

This document is intended for, and should only be read by, those persons to whom it is addressed. Its contents are confidential and if you have received this message in error, please delete it. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without our prior written consent is strictly prohibited.

Any views expressed in this message are those of the individual sender, and do not necessarily represent the position of Cubic Transportation Systems (ITMS) Limited (‘CTS’). Furthermore CTS does not authorise or use e-mail for official contractual correspondence. Nothing received in e-mail has any contractual validity.

CTSL and each legal entity in Cubic Corporation reserve the right to monitor all e-mail communications through its networks.

Registered Office:
Cubic Transportation Systems Ltd
AFC House
Honeycrock Lane
Salfords
Surrey
RH1 5LA
United Kingdom

Registered in England under number 8498086

________________________________

2

On Fri, Aug 7, 2015 at 10:28 AM, Kirk, Victor <VICTOR.KIRK@anonymised.com> wrote:

Hi,

We have a polygon layer used as a backdrop for a WMS based layer group.
The layer contains the outlines of the UK, one of which is the mainland and
is obviously quite large. I have generalised versions of the geometries to
use when zoomed out, however when zoomed in, encoding the whole mainland
for every tile introduces a bottleneck and requests quickly build up and
time out as every request. The layer group contains around 20 different
layers, without the polygon layer the group renders very quickly add the
layer back in and it becomes very unsresposive.

The layer is used to provide the coast and a background colour for land.
I'm looking at "chopping" up the polygons on a grid to create smaller
geometries that can be used to provide the background colour and extract
the coast into a number of smaller lines.

Yes, this is the standard trick to handle large polygons.

But before I do this I was wondering if there was some way I could get
geoserver to cache the features for this layer in memory rather than having
to hit the database for every request.

No, there is no facility to keep features in memory, we do no data caching
whatsoever.

Full layer caching would be relatively easy to implement though,
partial/usage sensitive one would be harder (only keep in memory the hot
areas),
both would require either dedicated resources to work it, or funding to one
of the commercial support providers towards implementing that specific
feature.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

AFAIK there is no way to cache a layer in memory. This would be a useful feature to add to GeoServer.

Perhaps you could cache the geometry as a shapefile - which should provide fastest performance off disk? It would be interesting to see what the performance boost is at any rate. I also wonder if clipping small parts out of a very large geometry is causing a performance hit.

···

On Fri, Aug 7, 2015 at 1:28 AM, Kirk, Victor <VICTOR.KIRK@anonymised.com> wrote:

Hi,

We have a polygon layer used as a backdrop for a WMS based layer group. The layer contains the outlines of the UK, one of which is the mainland and is obviously quite large. I have generalised versions of the geometries to use when zoomed out, however when zoomed in, encoding the whole mainland for every tile introduces a bottleneck and requests quickly build up and time out as every request. The layer group contains around 20 different layers, without the polygon layer the group renders very quickly add the layer back in and it becomes very unsresposive.

The layer is used to provide the coast and a background colour for land. I’m looking at “chopping” up the polygons on a grid to create smaller geometries that can be used to provide the background colour and extract the coast into a number of smaller lines.

But before I do this I was wondering if there was some way I could get geoserver to cache the features for this layer in memory rather than having to hit the database for every request.

Regards, Vic

This document is intended for, and should only be read by, those persons to whom it is addressed. Its contents are confidential and if you have received this message in error, please delete it. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without our prior written consent is strictly prohibited.

Any views expressed in this message are those of the individual sender, and do not necessarily represent the position of Cubic Transportation Systems (ITMS) Limited (‘CTS’). Furthermore CTS does not authorise or use e-mail for official contractual correspondence. Nothing received in e-mail has any contractual validity.

CTSL and each legal entity in Cubic Corporation reserve the right to monitor all e-mail communications through its networks.

Registered Office:
Cubic Transportation Systems Ltd
AFC House
Honeycrock Lane
Salfords
Surrey
RH1 5LA
United Kingdom

Registered in England under number 8498086

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

4

On Sun, Aug 9, 2015 at 5:19 PM, Martin Davis <mtnclimb@anonymised.com> wrote:

AFAIK there is no way to cache a layer in memory. This would be a useful
feature to add to GeoServer.

Perhaps you could cache the geometry as a shapefile - which should provide
fastest performance off disk? It would be interesting to see what the
performance boost is at any rate. I also wonder if clipping small parts
out of a very large geometry is causing a performance hit.

We have a fast rectangular clipping algorithm geared towards rendering that
is significantly faster than
JTS intersection (as we don't require the result to be topologically valid,
which greatly simplifies things).

It could still be that it's clipping, mind, but there are other possible
issues in play, besides the
obvious data transfer cost, which is going to be high regardless,
if the DB is postgis, loading from tables with large polygons might result
in the spatial index
not being used at all:

http://postgis.net/docs/performance_tips.html#small_tables_large_objects

Other spatial databases might have their own specific issues against
geometries with large number of vertices.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Thanks for the comments, a few things for me to investigate, I’ll try using a shape file and also add some metrics to the clipping (I’m already running a slightly modified version of geoserver).

Also glad to hear it’s common to chop up large polygons like I was thinking.

Kind Regards, Vic

···

On Sun, Aug 9, 2015 at 5:19 PM, Martin Davis <mtnclimb@anonymised.com.> wrote:

AFAIK there is no way to cache a layer in memory. This would be a useful feature to add to GeoServer.

Perhaps you could cache the geometry as a shapefile - which should provide fastest performance off disk? It would be interesting to see what the performance boost is at any rate. I also wonder if clipping small parts out of a very large geometry is causing a performance hit.

We have a fast rectangular clipping algorithm geared towards rendering that is significantly faster than
JTS intersection (as we don’t require the result to be topologically valid, which greatly simplifies things).

It could still be that it’s clipping, mind, but there are other possible issues in play, besides the
obvious data transfer cost, which is going to be high regardless,
if the DB is postgis, loading from tables with large polygons might result in the spatial index
not being used at all:

http://postgis.net/docs/performance_tips.html#small_tables_large_objects

Other spatial databases might have their own specific issues against geometries with large number of vertices.

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

6

Just to follow up on this in case anyone finds this in a search.

Switching to a shape file improved the response quite a bit, however spliting the data into smaller polygons and the coast into multiple line geometries was significantly better. All in all an average of 1500ms compared to 20ms request time (shape file method was around 200ms).

···

On Sun, Aug 9, 2015 at 5:19 PM, Martin Davis <mtnclimb@anonymised.com.> wrote:

AFAIK there is no way to cache a layer in memory. This would be a useful feature to add to GeoServer.

Perhaps you could cache the geometry as a shapefile - which should provide fastest performance off disk? It would be interesting to see what the performance boost is at any rate. I also wonder if clipping small parts out of a very large geometry is causing a performance hit.

We have a fast rectangular clipping algorithm geared towards rendering that is significantly faster than
JTS intersection (as we don’t require the result to be topologically valid, which greatly simplifies things).

It could still be that it’s clipping, mind, but there are other possible issues in play, besides the
obvious data transfer cost, which is going to be high regardless,
if the DB is postgis, loading from tables with large polygons might result in the spatial index
not being used at all:

http://postgis.net/docs/performance_tips.html#small_tables_large_objects

Other spatial databases might have their own specific issues against geometries with large number of vertices.

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.