[Geoserver-users] Seeking advice for large amount of layers

GeoServer GeoServer User
1

Hi all,

My team is using Geoserver to power a web app that shows maps of
projected climate change. We are currently running into issues that I
believe are related to the large amount of layers. I was wondering if
other users had ideas on how to make it more efficient and stable.

Our data is originally in Zarr datasets. It could be converted to
netCDF, but we had too many issues with the Rotated Pole projection. As
we added the reprojection step, we decided to save everything in
GeoTIFFs it seemed easier to handle.

The data has many facets : dataset (3), indicator (22), scenario (3),
season (5), stat (4), ensemble percentile (5), time period (15).

This list is somewhat simplified for the question. All in all, we
currently have 50 000 such geotiffs and as many layers in Geoserver.
That number is expected to grow a lot in the near future.

I see the multi-dimensional support in Geoserver, but in our case that
only covers the time period. Is there a format where we could combine
some layers together to have fewer stores on Geoserver ? I tried using
a GeoPackage with multiple "subdatasets", but Geoserver failed to parse
it.

We currently provision the server using the REST API. We mount the data
in a folder local to the server and make API calls to create each
layer/store. On the production server, this currently takes around 2h.
It's very slow. Is it possible to pre-generate the catalog offline ?

We haven't had performance issue for the WMS service for now, we don't
have that many users. Most difficulties arise in the data provisionning
step.

Thanks!

--
Pascal Bourgault
Ouranos, Montréal, Qc

2

We had the exact same type of problem. 1:1 GeoTIFF to layer.

We switched to using raster mosaics with a PostGIS mosaic index. We now have 1 store, 1 layer for each layer type.

We also had a ridiculous number of SLD files that were in the GeoServer directory and were therefore being managed by GeoServer. We moved this out of the folder and that also greatly improved performance.

Regards,
Richard

-----Original Message-----
From: Bourgault , Pascal via Geoserver-users <geoserver-users@lists.sourceforge.net>
Sent: Thursday, April 11, 2024 12:03 PM
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Seeking advice for large amount of layers

Hi all,

My team is using Geoserver to power a web app that shows maps of projected climate change. We are currently running into issues that I believe are related to the large amount of layers. I was wondering if other users had ideas on how to make it more efficient and stable.

Our data is originally in Zarr datasets. It could be converted to netCDF, but we had too many issues with the Rotated Pole projection. As we added the reprojection step, we decided to save everything in GeoTIFFs it seemed easier to handle.

The data has many facets : dataset (3), indicator (22), scenario (3), season (5), stat (4), ensemble percentile (5), time period (15).

This list is somewhat simplified for the question. All in all, we currently have 50 000 such geotiffs and as many layers in Geoserver.
That number is expected to grow a lot in the near future.

I see the multi-dimensional support in Geoserver, but in our case that only covers the time period. Is there a format where we could combine some layers together to have fewer stores on Geoserver ? I tried using a GeoPackage with multiple "subdatasets", but Geoserver failed to parse it.

We currently provision the server using the REST API. We mount the data in a folder local to the server and make API calls to create each layer/store. On the production server, this currently takes around 2h.
It's very slow. Is it possible to pre-generate the catalog offline ?

We haven't had performance issue for the WMS service for now, we don't have that many users. Most difficulties arise in the data provisionning step.

Thanks!

--
Pascal Bourgault
Ouranos, Montréal, Qc

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

I am curious why moving the SLD files out of geoserver helped performance? What part was slow? Did you replace them with GetMap with SLD=url parameter?

···

–
Jody Garnett

4

Dear Pascal,
you are missing a bit of your analysis.

ImageMosaic can handle multiple dimensions, not just time.
You can create custom dimensions for additional parameters and even use CQL_FILTERS with or without dimensions to filter further.
Just to give you an example, we tend to create large mosaics of Drone data which can be filtered by time, flightUUID, sensor and the like.

Going down the route of having millions of layers will become a problem in the long run so you need to find a better solution.
I would suggest having a look at this document to get more info.

On a separate note, supporting ZARR and your custom projection might be the best route in the longer term but this will require development.

···

Regards,

Simone Giannecchini

Online training classes for GeoNode, GeoServer and MapStore from the experts!
Visit https://www.geosolutionsgroup.com/professional-training/ for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director GeoSolutions Italy
President GeoSolutions USA

phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928
US: +1 (845) 547-7905

https://www.geosolutionsgroup.com
https://twitter.com/geosolutions_it

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

5

Thanks a lot everyone.

I didn't know about the CQL_FILTERS parameter on WMS requests. Thanks
Simone for the very detailed document going through exactly the issues
we are having!

As for the custom projection, our choice of re-projection was not only
taken because of our difficulties with Geoserver, but mostly because
our collaborators never know how to deal with Rotated Pole...

And for SLDs, I see many interesting options in the document. But I
also see that Geoserver supports ncWMS's extensions, which might be
just enough for our simple colormaps usecase.

Thanks!
--
Pascal

Le vendredi 12 avril 2024 à 11:44 +0200, Simone Giannecchini a écrit :

Dear Pascal,
you are missing a bit of your analysis.

ImageMosaic can handle _multiple_ dimensions, not just time.
You can create custom dimensions for additional parameters and even
use CQL_FILTERS with or without dimensions to filter further.
Just to give you an example, we tend to create large mosaics of Drone
data which can be filtered by time, flightUUID, sensor and the like.

Going down the route of having millions of layers will become a
problem in the long run so you need to find a better solution.
I would suggest having a look at this document to get more info.

On a separate note, supporting ZARR and your custom projection might
be the best route in the longer term but this will require
development.

Regards,
Simone Giannecchini

Online training classes for GeoNode, GeoServer and MapStore from the
experts!
Visit https://www.geosolutionsgroup.com/professional-training/ for
more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director GeoSolutions Italy
President GeoSolutions USA

phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928
US: +1 (845) 547-7905

https://www.geosolutionsgroup.com
https://twitter.com/geosolutions_it

-------------------------------------------------------
This email is intended only for the person or entity to which it is
addressed and may contain information that is privileged,
confidential or otherwise protected from disclosure. We remind that -
as provided by European Regulation 2016/679 “GDPR” - copying,
dissemination or use of this e-mail or the information herein by
anyone other than the intended recipient is prohibited. If you have
received this email by mistake, please notify us immediately by
telephone or e-mail.

On Thu, Apr 11, 2024 at 7:40 PM Bourgault , Pascal via Geoserver-
users <geoserver-users@lists.sourceforge.net> wrote:
> Hi all,
>
> My team is using Geoserver to power a web app that shows maps of
> projected climate change. We are currently running into issues that
> I
> believe are related to the large amount of layers. I was wondering
> if
> other users had ideas on how to make it more efficient and stable.
>
> Our data is originally in Zarr datasets. It could be converted to
> netCDF, but we had too many issues with the Rotated Pole
> projection. As
> we added the reprojection step, we decided to save everything in
> GeoTIFFs it seemed easier to handle.
>
> The data has many facets : dataset (3), indicator (22), scenario
> (3),
> season (5), stat (4), ensemble percentile (5), time period (15).
>
> This list is somewhat simplified for the question. All in all, we
> currently have 50 000 such geotiffs and as many layers in
> Geoserver.
> That number is expected to grow a lot in the near future.
>
> I see the multi-dimensional support in Geoserver, but in our case
> that
> only covers the time period. Is there a format where we could
> combine
> some layers together to have fewer stores on Geoserver ? I tried
> using
> a GeoPackage with multiple "subdatasets", but Geoserver failed to
> parse
> it.
>
> We currently provision the server using the REST API. We mount the
> data
> in a folder local to the server and make API calls to create each
> layer/store. On the production server, this currently takes around
> 2h.
> It's very slow. Is it possible to pre-generate the catalog offline
> ?
>
> We haven't had performance issue for the WMS service for now, we
> don't
> have that many users. Most difficulties arise in the data
> provisionning
> step.
>
> Thanks!
>
> --
> Pascal Bourgault
> Ouranos, Montréal, Qc
>
>
>
> _______________________________________________
> Geoserver-users mailing list
>
> Please make sure you read the following two resources before
> posting to this list:
> - Earning your support instead of buying it, but Ian Turton:
> http://www.ianturton.com/talks/foss4g.html#/
> - The GeoServer user list posting guidelines:
> http://geoserver.org/comm/userlist-guidelines.html
>
> If you want to request a feature or an improvement, also see this:
> https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
>
>
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

6

:+1:

···

Regards,

Simone Giannecchini

Online training classes for GeoNode, GeoServer and MapStore from the experts!
Visit https://www.geosolutionsgroup.com/professional-training/ for more information.

Ing. Simone Giannecchini
@simogeo
Founder/Director GeoSolutions Italy
President GeoSolutions USA

phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928
US: +1 (845) 547-7905

https://www.geosolutionsgroup.com
https://twitter.com/geosolutions_it

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.