[Geoserver-users] different layers at different zoom levels in WMS

Hi all,

Here goes another question :slight_smile:

I have a very big database with more than 50 million features stored, only points. I am playing visualizing this through WMS but as you can imagine it gets impossible when retrieving a picture of the whole world because the 50 million points must be taken from PostGIS and processed in order to create the map.

Then I started considering some kind of clustering. Something like grouping the points that are near in clusters and present them as a single point. I am still considering the clustering technology, well I am surfing internet looking for possibilities. Then when I get to the point that I have different features to present at different zooms levels, how can I force geoserver to produce different maps depending on this zoom level? I can imagine that can be done using SLD rules or something like this no? Do you know any place where I can find more info about this?

I suppose is the same as some WMS services providing satellite images that they provide you different pictures depending on the zoom level.

Any help on that?

Thanks.

Javier.

I’ve been looking for a solution to this problem as well but I can’t find anything in the documentation that suggests this is possible… Can any experts share their experiences with running geoserver in production with large datasets?

Corey

On 1/27/06, Javier de la Torre <jatorre@anonymised.com> wrote:

Hi all,

Here goes another question :slight_smile:

I have a very big database with more than 50 million features stored,
only points. I am playing visualizing this through WMS but as you can
imagine it gets impossible when retrieving a picture of the whole
world because the 50 million points must be taken from PostGIS and
processed in order to create the map.

Then I started considering some kind of clustering. Something like
grouping the points that are near in clusters and present them as a
single point. I am still considering the clustering technology, well
I am surfing internet looking for possibilities. Then when I get to
the point that I have different features to present at different
zooms levels, how can I force geoserver to produce different maps
depending on this zoom level? I can imagine that can be done using
SLD rules or something like this no? Do you know any place where I
can find more info about this?

I suppose is the same as some WMS services providing satellite images
that they provide you different pictures depending on the zoom level.

Any help on that?

Thanks.

Javier.


This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642


Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

This can easily be done with SLDs.
They allow for many layers, and all you have to do is specify some rules with zoom levels to render different layers.

Often you will see several layers representing the same dataset: very small and simple "zoomed out" layers, mid range layers, and the precise "zoomed in" layers.

Here is a tutorial that covers SLDs and goes into how to use rules:
http://docs.codehaus.org/display/GEOSDOC/SLD+Intro+Tutorial
This tutorial talks about more advanced rules, such as zoom levels:
http://docs.codehaus.org/display/GEOSDOC/ComplexLabelingExample

Here is the rule you are looking for:

<Rule>
  <MaxScaleDenominator>52000000</MaxScaleDenominator>
  <MinScaleDenominator>800000</MinScaleDenominator>
  ...
  // render the appropriate layer
  ...
</Rule>
// next rules for next layer

Brent Owens
(The Open Planning Project)

Corey Puffalt wrote:

I've been looking for a solution to this problem as well but I can't find anything in the documentation that suggests this is possible... Can any experts share their experiences with running geoserver in production with large datasets?

Corey

On 1/27/06, *Javier de la Torre* <jatorre@anonymised.com <mailto:jatorre@anonymised.com>> wrote:

    Hi all,

    Here goes another question :slight_smile:

    I have a very big database with more than 50 million features stored,
    only points. I am playing visualizing this through WMS but as you can
    imagine it gets impossible when retrieving a picture of the whole
    world because the 50 million points must be taken from PostGIS and
    processed in order to create the map.

    Then I started considering some kind of clustering. Something like
    grouping the points that are near in clusters and present them as a
    single point. I am still considering the clustering technology, well
    I am surfing internet looking for possibilities. Then when I get to
    the point that I have different features to present at different
    zooms levels, how can I force geoserver to produce different maps
    depending on this zoom level? I can imagine that can be done using
    SLD rules or something like this no? Do you know any place where I
    can find more info about this?

    I suppose is the same as some WMS services providing satellite images
    that they provide you different pictures depending on the zoom level.

    Any help on that?

    Thanks.

    Javier.

    -------------------------------------------------------
    This SF.net email is sponsored by: Splunk Inc. Do you grep through
    log files
    for problems? Stop! Download the new AJAX search engine that makes
    searching your log files as easy as surfing the web. DOWNLOAD
    SPLUNK!
    http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
    <http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642&gt;
    _______________________________________________
    Geoserver-users mailing list
    Geoserver-users@lists.sourceforge.net
    <mailto:Geoserver-users@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-users

Thanks Brent!

I see how powerful SLD is :slight_smile:

The only problem I see is that if Geoserver is right now not passing filters in SLD to SQL statements to retrieve features.
That means that I can not set up a column in my database for zoom level, or scale, and then filter the features based on this property. I will have to create a table for every scale with different features no?
And then I have to generate a SLD document that just point to the correct layer based on the Max Min Scale Denominators no? How can I set up a rule in SLD that has a Max Min thing and a pointer to another layer? Is there something like UseLayer element or something like this?

Am I getting to complicate? Is there anybody that has created a featuretype with more than a million features in a database? How can you handle the representation of this data?

Thanks again,

Javier.

On 28/01/2006, at 1:31, Brent Owens wrote:

This can easily be done with SLDs.
They allow for many layers, and all you have to do is specify some rules with zoom levels to render different layers.

Often you will see several layers representing the same dataset: very small and simple "zoomed out" layers, mid range layers, and the precise "zoomed in" layers.

Here is a tutorial that covers SLDs and goes into how to use rules:
http://docs.codehaus.org/display/GEOSDOC/SLD+Intro+Tutorial
This tutorial talks about more advanced rules, such as zoom levels:
http://docs.codehaus.org/display/GEOSDOC/ComplexLabelingExample

Here is the rule you are looking for:

<Rule>
  <MaxScaleDenominator>52000000</MaxScaleDenominator>
  <MinScaleDenominator>800000</MinScaleDenominator>
  ...
  // render the appropriate layer
  ...
</Rule>
// next rules for next layer

Brent Owens
(The Open Planning Project)

Corey Puffalt wrote:

I've been looking for a solution to this problem as well but I can't find anything in the documentation that suggests this is possible... Can any experts share their experiences with running geoserver in production with large datasets?

Corey

On 1/27/06, *Javier de la Torre* <jatorre@anonymised.com <mailto:jatorre@anonymised.com>> wrote:

    Hi all,

    Here goes another question :slight_smile:

    I have a very big database with more than 50 million features stored,
    only points. I am playing visualizing this through WMS but as you can
    imagine it gets impossible when retrieving a picture of the whole
    world because the 50 million points must be taken from PostGIS and
    processed in order to create the map.

    Then I started considering some kind of clustering. Something like
    grouping the points that are near in clusters and present them as a
    single point. I am still considering the clustering technology, well
    I am surfing internet looking for possibilities. Then when I get to
    the point that I have different features to present at different
    zooms levels, how can I force geoserver to produce different maps
    depending on this zoom level? I can imagine that can be done using
    SLD rules or something like this no? Do you know any place where I
    can find more info about this?

    I suppose is the same as some WMS services providing satellite images
    that they provide you different pictures depending on the zoom level.

    Any help on that?

    Thanks.

    Javier.

    -------------------------------------------------------
    This SF.net email is sponsored by: Splunk Inc. Do you grep through
    log files
    for problems? Stop! Download the new AJAX search engine that makes
    searching your log files as easy as surfing the web. DOWNLOAD
    SPLUNK!
    http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
    <http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642&gt;
    _______________________________________________
    Geoserver-users mailing list
    Geoserver-users@lists.sourceforge.net
    <mailto:Geoserver-users@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-users

Hi Javier

You are correct that all of the SLD rules do not get passed to the SQL statement. This is a limitation of it right now. It does pass some information to the SQL statement, but the rest is used on the client side. This might be changing in the future (hopefully). I really want to see it introduced.
The scale element is taken care of by GeoServer, but I think it is done after the query is performed.
You can have several different layers of the same data, and perform on the client side which to query. It shouldn't be very difficult to do that based on the zoom value you have on the client side.

We have definitely created data sets with several million entries and tested it with GeoServer. Everything is great when using WFS, but WMS you have to handle more of it on the client side and request generalized layers when zoomed out.

Brent Owens
(The Open Planning Project)

Javier de la Torre wrote:

Thanks Brent!

I see how powerful SLD is :slight_smile:

The only problem I see is that if Geoserver is right now not passing filters in SLD to SQL statements to retrieve features.
That means that I can not set up a column in my database for zoom level, or scale, and then filter the features based on this property. I will have to create a table for every scale with different features no?
And then I have to generate a SLD document that just point to the correct layer based on the Max Min Scale Denominators no? How can I set up a rule in SLD that has a Max Min thing and a pointer to another layer? Is there something like UseLayer element or something like this?

Am I getting to complicate? Is there anybody that has created a featuretype with more than a million features in a database? How can you handle the representation of this data?

Thanks again,

Javier.

On 28/01/2006, at 1:31, Brent Owens wrote:

This can easily be done with SLDs.
They allow for many layers, and all you have to do is specify some rules with zoom levels to render different layers.

Often you will see several layers representing the same dataset: very small and simple "zoomed out" layers, mid range layers, and the precise "zoomed in" layers.

Here is a tutorial that covers SLDs and goes into how to use rules:
http://docs.codehaus.org/display/GEOSDOC/SLD+Intro+Tutorial
This tutorial talks about more advanced rules, such as zoom levels:
http://docs.codehaus.org/display/GEOSDOC/ComplexLabelingExample

Here is the rule you are looking for:

<Rule>
    <MaxScaleDenominator>52000000</MaxScaleDenominator>
    <MinScaleDenominator>800000</MinScaleDenominator>
    ...
    // render the appropriate layer
    ...
</Rule>
// next rules for next layer

Brent Owens
(The Open Planning Project)

Corey Puffalt wrote:

I've been looking for a solution to this problem as well but I can't find anything in the documentation that suggests this is possible... Can any experts share their experiences with running geoserver in production with large datasets?

Corey

On 1/27/06, *Javier de la Torre* <jatorre@anonymised.com <mailto:jatorre@anonymised.com>> wrote:

    Hi all,

    Here goes another question :slight_smile:

    I have a very big database with more than 50 million features stored,
    only points. I am playing visualizing this through WMS but as you can
    imagine it gets impossible when retrieving a picture of the whole
    world because the 50 million points must be taken from PostGIS and
    processed in order to create the map.

    Then I started considering some kind of clustering. Something like
    grouping the points that are near in clusters and present them as a
    single point. I am still considering the clustering technology, well
    I am surfing internet looking for possibilities. Then when I get to
    the point that I have different features to present at different
    zooms levels, how can I force geoserver to produce different maps
    depending on this zoom level? I can imagine that can be done using
    SLD rules or something like this no? Do you know any place where I
    can find more info about this?

    I suppose is the same as some WMS services providing satellite images
    that they provide you different pictures depending on the zoom level.

    Any help on that?

    Thanks.

    Javier.

    -------------------------------------------------------
    This SF.net email is sponsored by: Splunk Inc. Do you grep through
    log files
    for problems? Stop! Download the new AJAX search engine that makes
    searching your log files as easy as surfing the web. DOWNLOAD
    SPLUNK!
    http://sel.as-us.falkag.net/sel? cmd=lnk&kid=103432&bid=230486&dat=121642
    <http://sel.as-us.falkag.net/sel? cmd=lnk&kid=103432&bid=230486&dat=121642>
    _______________________________________________
    Geoserver-users mailing list
    Geoserver-users@lists.sourceforge.net
    <mailto:Geoserver-users@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-users

Hi Brent,

You can have several different layers of the same data, and perform on the client side which to query. It shouldn't be very difficult to do that based on the zoom value you have on the client side.

But, how can I define different layers on the same data? You mean creating different Data types based on the same database table and apply different rules in the SLD document by default? So, for example, in my table I will have a column called zoom_level to indicate that this record should only be shown in this zoom level, then I create a featuretype called my_layer_at_zoom12 and create an SLD document for it that filter based on the zoom_level property.
Am I right with this?
If so I still have the problem that because filters in SLD documents are post-processed then to generate an image geoserver will retrieve all features from the database.

Other possibility could be to use views in PostgreSQL using this zoom_level and configuring feature types based on them, no?

We have definitely created data sets with several million entries and tested it with GeoServer. Everything is great when using WFS, but WMS you have to handle more of it on the client side and request generalized layers when zoomed out.

I see, but if you go the WFS way when presenting large datasets you have to create pretty robust clients that can handle a lot of features and draw them. And also you have the annoying problem that you don´t know if you are getting all features for your query or is just the limit that the user set in geoserver. And if you want to retrieve 10.000 features and the user limit them to 1.000 there is no way you will be able to get the other 9.000 no?

That´s why I thought that WMS was a better way to handle large datasets, all drawing is done in the server and only when the user click somewhere uses WFS to retrieve info about specific features.

Javier.

Comments inline:

Javier de la Torre wrote:

Hi Brent,

You can have several different layers of the same data, and perform on the client side which to query. It shouldn't be very difficult to do that based on the zoom value you have on the client side.

But, how can I define different layers on the same data? You mean creating different Data types based on the same database table and apply different rules in the SLD document by default? So, for example, in my table I will have a column called zoom_level to indicate that this record should only be shown in this zoom level, then I create a featuretype called my_layer_at_zoom12 and create an SLD document for it that filter based on the zoom_level property.
Am I right with this?
If so I still have the problem that because filters in SLD documents are post-processed then to generate an image geoserver will retrieve all features from the database.

Other possibility could be to use views in PostgreSQL using this zoom_level and configuring feature types based on them, no?

What I mean is, you can create several copies of your data, say 3 or 4, and generalize them into smaller, simpler, datasets. So your most zoomed in dataset is the origional, complex dataset (in its own table), and the most zoomed out dataset is the simplest one with the least amount of data.

When your client script makes a getMap request, it can look at how zoomed in you are, and request the different datasets (FeatureType): the simplest one for zoomed out, and the original dataset for when it is zoomed in. So in the end you will have several FeatureTypes of the data, some simpler than the others.
The scale element in SLD is really only useful for filtering before rendering, so things look nice; it is not useful for filtering data in the query.

To create the different datasets from the one dataset, you will need a tool such as FME. There are other tools out there that are free, but I cna't remember the names off the top of my head.

Filtering your data on your 'zoom level' column is a good idea, but unfortunately it will be done after all the data has been selected in the query. Using views is a solution for this problem.

We have definitely created data sets with several million entries and tested it with GeoServer. Everything is great when using WFS, but WMS you have to handle more of it on the client side and request generalized layers when zoomed out.

I see, but if you go the WFS way when presenting large datasets you have to create pretty robust clients that can handle a lot of features and draw them. And also you have the annoying problem that you don´t know if you are getting all features for your query or is just the limit that the user set in geoserver. And if you want to retrieve 10.000 features and the user limit them to 1.000 there is no way you will be able to get the other 9.000 no?

That´s why I thought that WMS was a better way to handle large datasets, all drawing is done in the server and only when the user click somewhere uses WFS to retrieve info about specific features.

It is better to do the rendering on the server side with WMS, so the data passed to the client is minimal. And so you don't have to write your own renderer.
If you are worred about your users modifying the number of features returned, just don't let them specify the number. You can always set the max_features_returned value to be larger than your dataset and will get every row returned, if that is what was selected in the query.

Javier.

Brent Owens
(The Open Planning Project)

Hi Brent,

To create the different datasets from the one dataset, you will need a tool such as FME. There are other tools out there that are free, but I cna't remember the names off the top of my head.

I am doing something by hand, but I would be really happy if I can find any free tool to do clustering or splits of geographical datasets. Specially something that can connect to PostGIS and work directly with the data there.

Javier.