[Geoserver-users] Speeding up GeoWebCache seeding process

GeoServer GeoServer User
1

Hi,

Can you share the data and CSS?

-Jukka Rahkonen-

···

Nachtigall, Jens wrote:

Hi again,

I really do not understand why it takes weeks for a simple 100 MB Shapefile with just LineStrings in Germany to render. 99.99 % of all tiles are just empty/transparent PNGs anyways, rest is just some red lines… I googled and found http://gis.stackexchange.com/questions/20712/geoserver-wms-tile-rendering-is-too-slow

  1. Would it make a difference to add a spatial index to the Shapefile? (I did not check if it has one, I would assume that GeoServer just adds one if necessary – how can I check if this has been done?)

  2. Also, we use the GeoServer-CSS extension. But the style is as simple as this:

  • {

fill: #e31a1c;

fill-opacity: 0.27;

stroke: #e31a1c;

stroke-width: 1;

stroke-opacity: 1;

}

Anyway, can this be related to the long seeding times?

  1. Maybe some kind of metatiling can be used to speed things up?

For OSM rendering with mapnik for same extent and much more features, it took just a few days. So I really do not get it, why it takes weeks for a damn simply Shapefile.

Ideas welcome…

Jens

2

Hi,
having no data, I’ve downloaded a shapefile of germany roads from http://www.mapcruzin.com/free-germany-arcgis-maps-shapefiles.htm,
300MB worth of shp file, and applied the default line style… and realized I did something that makes no sense, the output map looks like
this:

Well, of course it takes a lot of time to paint this map, it’s painting 2.7 million roads in a small space.
Moreover, GWC starts seeding from the top, from the slowest tiles, so the estimate is larger that
reality (mostly because of this style that is way more expensive to draw when zoomed out than
when zoomed in, in normal maps like OSM the cost is not same, but similar, because not all
data is painted when zoomed out).

That said, during the time it took me to write this mail I let GWC run and seed the cache with 4 threads
on a old core i7 820, on Linux, OpenJDK 7, from level 0 to 15, after 10 minutes it reached zoom level 14 and the
tasks report:

​Not too bad I’d say? If this is multiply by 4 to get to level 16 and 17, it would take

4 hours to do zoom level 16, and 16 hours to do zoom level 17… so if I had done this against
zoom level 0 to 17 it would have taken less than 24 hours, with a larger shapefile, and
with a silly style.

Cheers
Andrea
​

···

On Tue, Mar 31, 2015 at 3:38 PM, Rahkonen Jukka (MML) <jukka.rahkonen@anonymised.com> wrote:

Hi,

Can you share the data and CSS?

-Jukka Rahkonen-

Nachtigall, Jens wrote:

Hi again,

I really do not understand why it takes weeks for a simple 100 MB Shapefile with just LineStrings in Germany to render. 99.99 % of all tiles are just empty/transparent PNGs anyways, rest is just some red lines… I googled and found http://gis.stackexchange.com/questions/20712/geoserver-wms-tile-rendering-is-too-slow

  1. Would it make a difference to add a spatial index to the Shapefile? (I did not check if it has one, I would assume that GeoServer just adds one if necessary – how can I check if this has been done?)

  2. Also, we use the GeoServer-CSS extension. But the style is as simple as this:

  • {

fill: #e31a1c;

fill-opacity: 0.27;

stroke: #e31a1c;

stroke-width: 1;

stroke-opacity: 1;

}

Anyway, can this be related to the long seeding times?

  1. Maybe some kind of metatiling can be used to speed things up?

For OSM rendering with mapnik for same extent and much more features, it took just a few days. So I really do not get it, why it takes weeks for a damn simply Shapefile.

Ideas welcome…

Jens

Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

Hi

I think the bottleneck is not the rendering process but writing the tiles to disk.

To improve the process of writing, we should used instead of the png files, an equivalent ArcGIS Compact Cache, MbTiles or GeoPackage…

The rendering process (I do not know what is used) can be expected to go through a similar strategy to the vector tiling: i.e. store vectors in an R-Tree and used the R-Tree to build the tiles.

Best regards

Selezione_015.png

nurc-roads.jpg

···

2015-03-31 16:28 GMT+02:00 Andrea Aime <andrea.aime@anonymised.com>:

Hi,
having no data, I’ve downloaded a shapefile of germany roads from http://www.mapcruzin.com/free-germany-arcgis-maps-shapefiles.htm,
300MB worth of shp file, and applied the default line style… and realized I did something that makes no sense, the output map looks like
this:

Well, of course it takes a lot of time to paint this map, it’s painting 2.7 million roads in a small space.
Moreover, GWC starts seeding from the top, from the slowest tiles, so the estimate is larger that
reality (mostly because of this style that is way more expensive to draw when zoomed out than
when zoomed in, in normal maps like OSM the cost is not same, but similar, because not all
data is painted when zoomed out).

That said, during the time it took me to write this mail I let GWC run and seed the cache with 4 threads
on a old core i7 820, on Linux, OpenJDK 7, from level 0 to 15, after 10 minutes it reached zoom level 14 and the
tasks report:

​Not too bad I’d say? If this is multiply by 4 to get to level 16 and 17, it would take

4 hours to do zoom level 16, and 16 hours to do zoom level 17… so if I had done this against
zoom level 0 to 17 it would have taken less than 24 hours, with a larger shapefile, and
with a silly style.

Cheers
Andrea
​

Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

On Tue, Mar 31, 2015 at 3:38 PM, Rahkonen Jukka (MML) <jukka.rahkonen@anonymised.com> wrote:

Hi,

Can you share the data and CSS?

-Jukka Rahkonen-

Nachtigall, Jens wrote:

Hi again,

I really do not understand why it takes weeks for a simple 100 MB Shapefile with just LineStrings in Germany to render. 99.99 % of all tiles are just empty/transparent PNGs anyways, rest is just some red lines… I googled and found http://gis.stackexchange.com/questions/20712/geoserver-wms-tile-rendering-is-too-slow

  1. Would it make a difference to add a spatial index to the Shapefile? (I did not check if it has one, I would assume that GeoServer just adds one if necessary – how can I check if this has been done?)

  2. Also, we use the GeoServer-CSS extension. But the style is as simple as this:

  • {

fill: #e31a1c;

fill-opacity: 0.27;

stroke: #e31a1c;

stroke-width: 1;

stroke-opacity: 1;

}

Anyway, can this be related to the long seeding times?

  1. Maybe some kind of metatiling can be used to speed things up?

For OSM rendering with mapnik for same extent and much more features, it took just a few days. So I really do not get it, why it takes weeks for a damn simply Shapefile.

Ideas welcome…

Jens

Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

On Tue, Mar 31, 2015 at 7:29 PM, xavier lhomme <lhomme.xavier@anonymised.com>
wrote:

Hi
I think the bottleneck is not the rendering process but writing the tiles
to disk.

The reporter notes that the CPUs are pegged, so in his case, it does not
look like the disk is much of an issue.
In my little test it was hardly a problem too.

To improve the process of writing, we should used instead of the png
files, an equivalent ArcGIS Compact Cache, MbTiles or GeoPackage...

Yep, both well known direction for improvements, both in need of
sponsorship :slight_smile:

The rendering process (I do not know what is used) can be expected to go
through a similar strategy to the vector tiling: i.e. store vectors in an
R-Tree and used the R-Tree to build the tiles.

Assuming you can keep the data in memory, which is not always the case...
in general GeoServer tries to stay the hell away from being memory bound,
but
an application that is dedicated to tile seeding could try to do that.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------