[Geoserver-users] AOP to modify the WMS request?

GeoServer GeoServer User
1

Hi,

I am still finding my feet with Geoserver and PostGIS, but have both built
and running from source now. My motivation for doing this is to find a
fairly simply / standards comliant way of serving WMS for 100,000,000
specimen/observation records within the Biodiversity field (www.gbif.org).

I have tested postgis and mysql behind geoserver each serving 1 million
records on the same schema and both perform pretty similarly (although
mysql was over 10x quicker to populate with records).

However, at a global view, obviously this is a daft approach as individual
points at this scale is meaningles - I already have the data in collated
counts at 1x1 degree cells, so anything with a bounding box search greater
than say 20x20 degrees, I wish to return from the counts table (e.g. 1x1
degree cells with varying colours based on the count within the cell -
using rules in the SLD), otherwise I would return the individual records
as points when zoomed in enough.

So - is a suitable approach for me to set up 2 datastores against the 2
tables, and then intercept the request using an AOP interceptor and modify
it based on the bounding box?

What is my alternative? - implement my own datastore?

Or how else can you swap layers based on zoom level?

Alternatively, how do people generally hook in an existing DAO / Service
layer - implement the data store?

Thanks all - I'd appreciate any thoughts / suggestions / pointing out the
obvious - I am new to this!

Tim

2

Tim,

Not sure if you've hit on this suggestion already, but a much much *much* lower tech way of doing this (no need to write any interceptors, datastores or anything!) would be to just define two SLDs, each relevant to the right featuretype.

I.e:

1) low-res-polys = one feature type (connected to your generalized data) with its own SLD. Put a MinScaleDenominator in this SLD, so that it turns 'off' when you zoom in past its useful level.
2) high-res-points = one feature type (connected to your actual 100,000,000 records) with its own SLD. Put a MaxScaleDenominator in this SLD, so that it doesn't render at higher scales.

The last step is to set up a map viewer with BOTH layers ALWAYS turned on. As long as the minscaledenominator of the one lines up with the maxscaledenominator of the other, users will have a 'seamless' transition between the two...and you don't have to write any code! You just have to make a map view that occludes what's actually going on in the back-ground (two layers in geoserver -> one layer in the viewer)

The down-side of this approach is that geoserver will still willingly attempt to render 100,000,000 points with a custom SLD, if a user were to craft his/her own SLD and send it along. A more interesting question is whether you could code support for "disabling" custom SLDs on a layer-by-layer basis. That way you could disable any custom SLDs for your really big layer.

Am I repeating something that someone else already said? Is this too low-tech of a way to solve this problem?

--saul

-----Original Message-----
From: geoserver-users-bounces@lists.sourceforge.net on behalf of trobertson@anonymised.com
Sent: Fri 4/6/2007 4:41 PM
To: geoserver users
Subject: [Geoserver-users] AOP to modify the WMS request?

Hi,

I am still finding my feet with Geoserver and PostGIS, but have both built
and running from source now. My motivation for doing this is to find a
fairly simply / standards comliant way of serving WMS for 100,000,000
specimen/observation records within the Biodiversity field (www.gbif.org).

I have tested postgis and mysql behind geoserver each serving 1 million
records on the same schema and both perform pretty similarly (although
mysql was over 10x quicker to populate with records).

However, at a global view, obviously this is a daft approach as individual
points at this scale is meaningles - I already have the data in collated
counts at 1x1 degree cells, so anything with a bounding box search greater
than say 20x20 degrees, I wish to return from the counts table (e.g. 1x1
degree cells with varying colours based on the count within the cell -
using rules in the SLD), otherwise I would return the individual records
as points when zoomed in enough.

So - is a suitable approach for me to set up 2 datastores against the 2
tables, and then intercept the request using an AOP interceptor and modify
it based on the bounding box?

What is my alternative? - implement my own datastore?

Or how else can you swap layers based on zoom level?

Alternatively, how do people generally hook in an existing DAO / Service
layer - implement the data store?

Thanks all - I'd appreciate any thoughts / suggestions / pointing out the
obvious - I am new to this!

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

Farber, Saul (ENV) wrote:

Tim,

Not sure if you've hit on this suggestion already, but a much much *much* lower tech way of doing this (no need to write any interceptors, datastores or anything!) would be to just define two SLDs, each relevant to the right featuretype.

I.e:

1) low-res-polys = one feature type (connected to your generalized data) with its own SLD. Put a MinScaleDenominator in this SLD, so that it turns 'off' when you zoom in past its useful level.
2) high-res-points = one feature type (connected to your actual 100,000,000 records) with its own SLD. Put a MaxScaleDenominator in this SLD, so that it doesn't render at higher scales.

The last step is to set up a map viewer with BOTH layers ALWAYS turned on. As long as the minscaledenominator of the one lines up with the maxscaledenominator of the other, users will have a 'seamless' transition between the two...and you don't have to write any code! You just have to make a map view that occludes what's actually going on in the back-ground (two layers in geoserver -> one layer in the viewer)

The down-side of this approach is that geoserver will still willingly attempt to render 100,000,000 points with a custom SLD, if a user were to craft his/her own SLD and send it along. A more interesting question is whether you could code support for "disabling" custom SLDs on a layer-by-layer basis. That way you could disable any custom SLDs for your really big layer.

Am I repeating something that someone else already said? Is this too low-tech of a way to solve this problem?

It's just how I'd go about it. We also have basic layer grouping functionality to be able to refer to multiple layers by one name. http://docs.codehaus.org/display/GEOSDOC/What+is+the+Layer+Group+option

So make SLDs for both, and then just make a layer group, and give clients that name.

An option to turn off user defined SLDs per layer would be nice. Though at this point I know of know easy to use clients that leverage SLD, so someone would have to be trying pretty hard to do that. Another thought is to have maxFeatures also apply to the WMS, though I imagine one might want different values for what each will return.

Chris

--saul

-----Original Message-----
From: geoserver-users-bounces@lists.sourceforge.net on behalf of trobertson@anonymised.com
Sent: Fri 4/6/2007 4:41 PM
To: geoserver users
Subject: [Geoserver-users] AOP to modify the WMS request?
Hi,

I am still finding my feet with Geoserver and PostGIS, but have both built
and running from source now. My motivation for doing this is to find a
fairly simply / standards comliant way of serving WMS for 100,000,000
specimen/observation records within the Biodiversity field (www.gbif.org).

I have tested postgis and mysql behind geoserver each serving 1 million
records on the same schema and both perform pretty similarly (although
mysql was over 10x quicker to populate with records).

However, at a global view, obviously this is a daft approach as individual
points at this scale is meaningles - I already have the data in collated
counts at 1x1 degree cells, so anything with a bounding box search greater
than say 20x20 degrees, I wish to return from the counts table (e.g. 1x1
degree cells with varying colours based on the count within the cell -
using rules in the SLD), otherwise I would return the individual records
as points when zoomed in enough.

So - is a suitable approach for me to set up 2 datastores against the 2
tables, and then intercept the request using an AOP interceptor and modify
it based on the bounding box?

What is my alternative? - implement my own datastore?

Or how else can you swap layers based on zoom level?

Alternatively, how do people generally hook in an existing DAO / Service
layer - implement the data store?

Thanks all - I'd appreciate any thoughts / suggestions / pointing out the
obvious - I am new to this!

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Chris Holmes
The Open Planning Project
http://topp.openplans.org

4

Chris Holmes ha scritto:

An option to turn off user defined SLDs per layer would be nice. Though at this point I know of know easy to use clients that leverage SLD, so someone would have to be trying pretty hard to do that. Another thought is to have maxFeatures also apply to the WMS, though I imagine one might want different values for what each will return.

Heh, this is one topic that had me thinking, in fact. WFS uses maxFeatures as a way to prevent users from making queries that do returns too many features, so it's a kind of resource usage limitator.

WMS does not have the equivalent, and this is bad. We could certainly
apply the same reasoning, but I'd argue that WMS is better protected
by a time limit than a CPU limit. WFS operations are often botllenecked by the network, at least if the network is an internet connection (on a relatively fast machine we do generate GML at a 4MB/s rate).
A WMS operation can easily take minutes of 100% CPU usage without being
bottlenecked by anything else, and this of course is a problem, a
few concurrent requests like this and the server is basically under
a DOS attack.

On the bright side, the renderer provides events for each feature
renderer, so we could listen to them, compute the elapsed time, and
stop rendering there, with a few options on the output:
* returning the image at its current state, with a text on it
   stating it time out
* get back with a WMS exception
* not returning anything, just close the connection

In fact, the same could be implemented on the WFS protocol by using
a feature reader wrapper that gives up when a certain time elapsed.

Comments?
Cheers
Andrea

5

Hi Saul,

Thank you very much for this - shortly after posting I started pondering
how the layers are switched on the demo and then learnt about that
approach too - I think that is probably the best and proper way to go...

Thanks

Tim

Tim,

Not sure if you've hit on this suggestion already, but a much much *much*
lower tech way of doing this (no need to write any interceptors,
datastores or anything!) would be to just define two SLDs, each relevant
to the right featuretype.

I.e:

1) low-res-polys = one feature type (connected to your generalized data)
with its own SLD. Put a MinScaleDenominator in this SLD, so that it turns
'off' when you zoom in past its useful level.
2) high-res-points = one feature type (connected to your actual
100,000,000 records) with its own SLD. Put a MaxScaleDenominator in this
SLD, so that it doesn't render at higher scales.

The last step is to set up a map viewer with BOTH layers ALWAYS turned on.
As long as the minscaledenominator of the one lines up with the
maxscaledenominator of the other, users will have a 'seamless' transition
between the two...and you don't have to write any code! You just have to
make a map view that occludes what's actually going on in the back-ground
(two layers in geoserver -> one layer in the viewer)

The down-side of this approach is that geoserver will still willingly
attempt to render 100,000,000 points with a custom SLD, if a user were to
craft his/her own SLD and send it along. A more interesting question is
whether you could code support for "disabling" custom SLDs on a
layer-by-layer basis. That way you could disable any custom SLDs for your
really big layer.

Am I repeating something that someone else already said? Is this too
low-tech of a way to solve this problem?

--saul

-----Original Message-----
From: geoserver-users-bounces@lists.sourceforge.net on behalf of
trobertson@anonymised.com
Sent: Fri 4/6/2007 4:41 PM
To: geoserver users
Subject: [Geoserver-users] AOP to modify the WMS request?

Hi,

I am still finding my feet with Geoserver and PostGIS, but have both built
and running from source now. My motivation for doing this is to find a
fairly simply / standards comliant way of serving WMS for 100,000,000
specimen/observation records within the Biodiversity field (www.gbif.org).

I have tested postgis and mysql behind geoserver each serving 1 million
records on the same schema and both perform pretty similarly (although
mysql was over 10x quicker to populate with records).

However, at a global view, obviously this is a daft approach as individual
points at this scale is meaningles - I already have the data in collated
counts at 1x1 degree cells, so anything with a bounding box search greater
than say 20x20 degrees, I wish to return from the counts table (e.g. 1x1
degree cells with varying colours based on the count within the cell -
using rules in the SLD), otherwise I would return the individual records
as points when zoomed in enough.

So - is a suitable approach for me to set up 2 datastores against the 2
tables, and then intercept the request using an AOP interceptor and modify
it based on the bounding box?

What is my alternative? - implement my own datastore?

Or how else can you swap layers based on zoom level?

Alternatively, how do people generally hook in an existing DAO / Service
layer - implement the data store?

Thanks all - I'd appreciate any thoughts / suggestions / pointing out the
obvious - I am new to this!

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

6

That sounds perfect - thanks fro the link Chris

Farber, Saul (ENV) wrote:

Tim,

Not sure if you've hit on this suggestion already, but a much much
*much* lower tech way of doing this (no need to write any interceptors,
datastores or anything!) would be to just define two SLDs, each relevant
to the right featuretype.

I.e:

1) low-res-polys = one feature type (connected to your generalized data)
with its own SLD. Put a MinScaleDenominator in this SLD, so that it
turns 'off' when you zoom in past its useful level.
2) high-res-points = one feature type (connected to your actual
100,000,000 records) with its own SLD. Put a MaxScaleDenominator in
this SLD, so that it doesn't render at higher scales.

The last step is to set up a map viewer with BOTH layers ALWAYS turned
on. As long as the minscaledenominator of the one lines up with the
maxscaledenominator of the other, users will have a 'seamless'
transition between the two...and you don't have to write any code! You
just have to make a map view that occludes what's actually going on in
the back-ground (two layers in geoserver -> one layer in the viewer)

The down-side of this approach is that geoserver will still willingly
attempt to render 100,000,000 points with a custom SLD, if a user were
to craft his/her own SLD and send it along. A more interesting question
is whether you could code support for "disabling" custom SLDs on a
layer-by-layer basis. That way you could disable any custom SLDs for
your really big layer.

Am I repeating something that someone else already said? Is this too
low-tech of a way to solve this problem?

It's just how I'd go about it. We also have basic layer grouping
functionality to be able to refer to multiple layers by one name.
http://docs.codehaus.org/display/GEOSDOC/What+is+the+Layer+Group+option

So make SLDs for both, and then just make a layer group, and give
clients that name.

An option to turn off user defined SLDs per layer would be nice. Though
at this point I know of know easy to use clients that leverage SLD, so
someone would have to be trying pretty hard to do that. Another thought
is to have maxFeatures also apply to the WMS, though I imagine one might
want different values for what each will return.

Chris

--saul

-----Original Message-----
From: geoserver-users-bounces@lists.sourceforge.net on behalf of
trobertson@anonymised.com
Sent: Fri 4/6/2007 4:41 PM
To: geoserver users
Subject: [Geoserver-users] AOP to modify the WMS request?

Hi,

I am still finding my feet with Geoserver and PostGIS, but have both
built
and running from source now. My motivation for doing this is to find a
fairly simply / standards comliant way of serving WMS for 100,000,000
specimen/observation records within the Biodiversity field
(www.gbif.org).

I have tested postgis and mysql behind geoserver each serving 1 million
records on the same schema and both perform pretty similarly (although
mysql was over 10x quicker to populate with records).

However, at a global view, obviously this is a daft approach as
individual
points at this scale is meaningles - I already have the data in collated
counts at 1x1 degree cells, so anything with a bounding box search
greater
than say 20x20 degrees, I wish to return from the counts table (e.g. 1x1
degree cells with varying colours based on the count within the cell -
using rules in the SLD), otherwise I would return the individual records
as points when zoomed in enough.

So - is a suitable approach for me to set up 2 datastores against the 2
tables, and then intercept the request using an AOP interceptor and
modify
it based on the bounding box?

What is my alternative? - implement my own datastore?

Or how else can you swap layers based on zoom level?

Alternatively, how do people generally hook in an existing DAO / Service
layer - implement the data store?

Thanks all - I'd appreciate any thoughts / suggestions / pointing out
the
obvious - I am new to this!

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Chris Holmes
The Open Planning Project
http://topp.openplans.org

7

Hi All,

I have a DB table with a point and an integer of occurrences at that
point. Is it possible to have an SLD that has a circle which varies in
size depending on the count of occurrences at that point?

A newbie question I know... but struggling to find the syntax for SLDs.

Thanks for help

Tim

8

Yes, the symbol can vary in size based on an attribute (or even based on
some math you do with attributes) - take a look at the bottom half of
this page:
http://lyceum.massgis.state.ma.us/wiki/doku.php?id=wms:sld:point_example
_graduated&s=sld

9

hmm, I don't think it will work to do that currently, but it shouldn't
be too difficult to add that feature in. You can try though, I could
be wrong... try putting the
<ogc:PropertyName>field_name</ogc:PropertyName> tag in the radius
value.

On 4/9/07, trobertson@anonymised.com <trobertson@anonymised.com> wrote:

Hi All,

I have a DB table with a point and an integer of occurrences at that
point. Is it possible to have an SLD that has a circle which varies in
size depending on the count of occurrences at that point?

A newbie question I know... but struggling to find the syntax for SLDs.

Thanks for help

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Brent Owens

10

Thanks to Aleda and Brent for those quick replies!

Looks ideal Aleda.

Yes, the symbol can vary in size based on an attribute (or even based on
some math you do with attributes) - take a look at the bottom half of
this page:
http://lyceum.massgis.state.ma.us/wiki/doku.php?id=wms:sld:point_example
_graduated&s=sld

11

Brent Owens ha scritto:

hmm, I don't think it will work to do that currently, but it shouldn't
be too difficult to add that feature in. You can try though, I could
be wrong... try putting the
<ogc:PropertyName>field_name</ogc:PropertyName> tag in the radius
value.

It should be working, even with an expression, but I suggest to try both that, and setting up a set of rules that do filter the points based on the attribute value, and use a fixed (but different) size in each rule.
Performance wise, the results may be quite a bit different, but I'm unsure which would be the fastest. The expression will issue just one query, but will compute and create a resolved symbolizer per point, whilst the rule based solution may issue one query per rule, but
will for sure create as many resolved symbolizers as the number of the
rules, and the cache and reuse them instead of recomputing one new for
each point. Oh, if you're wondering, a resolved symbolizer is an object
that has a fixed color, a fixed stroke size, a fixed symbol and so on.
It's what you get when you apply a symbolizer against a feature.

Cheers
Andrea

12

Hi

Can someone please point out the error in this?

I am trying to scale the fill opacity - I have ranges of 1-325 on the count

Thanks

Tim

<CssParameter name="fill-opacity">
  <Divide>
    <ogc:PropertyName>count</ogc:PropertyName>
    <ogc:Literal>325</ogc:Literal>
  </Divide>
</CssParameter>

13

trobertson@anonymised.com ha scritto:

Hi

Can someone please point out the error in this?

I am trying to scale the fill opacity - I have ranges of 1-325 on the count

Thanks

Hum, according to the OGC filter 1.0 spec, it's Div, not Divide:

<xsd:element name="Div"
                type="ogc:BinaryOperatorType"
                substitutionGroup="ogc:expression"/>
I guess that should be it...

Cheers
Andrea

14

trobertson@anonymised.com ha scritto:

Hi

Can someone please point out the error in this?

I am trying to scale the fill opacity - I have ranges of 1-325 on the count

Thanks

Tim

<CssParameter name="fill-opacity">
  <Divide>
    <ogc:PropertyName>count</ogc:PropertyName>
    <ogc:Literal>325</ogc:Literal>
  </Divide>
</CssParameter>

I usually try to build SLD with a validating XML editor (this way I can
have auto-completion too), in my case I'm using Eclipse+WTP.

The following snippet does validate:

<?xml version="1.0" encoding="UTF-8"?>
<StyledLayerDescriptor version="1.0.0" xmlns="http://www.opengis.net/sld&quot; xmlns:ogc="http://www.opengis.net/ogc&quot;
   xmlns:xlink="http://www.w3.org/1999/xlink&quot; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
   xsi:schemaLocation="http://www.opengis.net/sld StyledLayerDescriptor.xsd http://www.w3.org/1999/xlink ../../xlink/1.0.0/xlinks.xsd http://www.opengis.net/ogc ../../filter/1.0.0/filter.xsd ">
   <NamedLayer>
     <Name>MyLayer</Name>
     <UserStyle>
       <FeatureTypeStyle>
         <Rule>
           <PolygonSymbolizer>
             <Fill>
               <CssParameter name="fill-opacity">
                 <ogc:Div>
                   <ogc:PropertyName>count</ogc:PropertyName>
                   <ogc:Literal>325</ogc:Literal>
                 </ogc:Div>
               </CssParameter>
             </Fill>
           </PolygonSymbolizer>
         </Rule>
       </FeatureTypeStyle>
     </UserStyle>
   </NamedLayer>
</StyledLayerDescriptor>

Cheers
Andrea