[Geoserver-devel] GeoWebCache and TIME

GeoServer GeoServer Developer
1

Hi, everybody.

I've got something hacked together so that my CoverageReader can handle TIME parameters, and I've tested it with wms calls through curl. Now I'm running into trouble when I try to hook it up to OpenLayers. It seems that GeoWebCache takes the incoming request and assembles a "fake" request from it with a fixed list of parameters. In GeoServerTileLayer.buildGetMap():

         params.put("SERVICE", "WMS");
         params.put("VERSION", "1.1.1");
         params.put("REQUEST", "GetMap");
         params.put("LAYERS", getName());
         params.put("SRS", srs);
         params.put("FORMAT", format);
         params.put("WIDTH", String.valueOf(width));
         params.put("HEIGHT", String.valueOf(height));
         params.put("BBOX", bbox.toString());

         params.put("EXCEPTIONS", GetMapRequest.SE_XML);
         params.put("STYLES", "");
         params.put("TRANSPARENT", "true");
         params.put(GWC_SEED_INTERCEPT_TOKEN, "true");

         Map<String, String> filteredParams = tile.getFullParameters();
         if (filteredParams.isEmpty()) {
             filteredParams = getDefaultParameterFilters();
         }
         params.putAll(filteredParams);

the last line could, I suppose, be used to add other parameters to the fake request, but tracing through the history of the tile object, I can't find any place where it might get added to tile.fullParameters.

Any ideas? How does this work for ImageMosaics? Thanks in advance.

2

On Sun, Feb 12, 2012 at 5:39 PM, John Armstrong <jrja@anonymised.com> wrote:

Hi, everybody.

I’ve got something hacked together so that my CoverageReader can handle
TIME parameters, and I’ve tested it with wms calls through curl. Now
I’m running into trouble when I try to hook it up to OpenLayers. It
seems that GeoWebCache takes the incoming request and assembles a “fake”
request from it with a fixed list of parameters. In
GeoServerTileLayer.buildGetMap():

params.put(“SERVICE”, “WMS”);
params.put(“VERSION”, “1.1.1”);
params.put(“REQUEST”, “GetMap”);
params.put(“LAYERS”, getName());
params.put(“SRS”, srs);
params.put(“FORMAT”, format);
params.put(“WIDTH”, String.valueOf(width));
params.put(“HEIGHT”, String.valueOf(height));
params.put(“BBOX”, bbox.toString());

params.put(“EXCEPTIONS”, GetMapRequest.SE_XML);
params.put(“STYLES”, “”);
params.put(“TRANSPARENT”, “true”);
params.put(GWC_SEED_INTERCEPT_TOKEN, “true”);

Map<String, String> filteredParams = tile.getFullParameters();
if (filteredParams.isEmpty()) {
filteredParams = getDefaultParameterFilters();
}
params.putAll(filteredParams);

the last line could, I suppose, be used to add other parameters to the
fake request, but tracing through the history of the tile object, I
can’t find any place where it might get added to tile.fullParameters.

Any ideas? How does this work for ImageMosaics? Thanks in advance.

It most likely doesn’t.

We actually did not check it, time + elevation would
require creating a separate tile cache for each different value passed in, which would
end up requiring a massive amount of disk space, so it’s a case I did not bother
to check when implementing time/elevation support.

Feel free to provide patches to make this case work (I suppose you have very
few values for time/elevation?)

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

3

On 02/12/2012 12:25 PM, Andrea Aime wrote:

On Sun, Feb 12, 2012 at 5:39 PM, John Armstrong <jrja@anonymised.com> wrote:

Any ideas? How does this work for ImageMosaics? Thanks in advance.

It most likely doesn’t.

Feel free to provide patches to make this case work (I suppose you have very
few values for time/elevation?)

I’ll see if I can come up with something; we’re looking at updating a layer daily and being able to retrieve timestamped tiles from a BigTable-like datastore. I don’t really know about typical use cases, but I wouldn’t be surprised if a user wanted to scroll through, say, a month’s worth of updates at various zoom levels. It all depends on what you mean by “very few”.

Thanks anyway.

4

This is how it should work. Let me know if it doesn’t:

  • Request comes in and is intercepted by CachingWebMapService
  • CachingWebMapService checks whether the request matches a configured tile cache.
    – if it doesn’t then request proceeds as normal, ultimately handled by the regular DefaultWebMapService, and gwc does not get into the mix anymore for the lifetime of the request
    – if it does match a tile cache (by bounds, format, layer, crs, etc), then the map is served by gwc.

So, in order for the CachingWebMapService to forward the request to gwc, the GWC mediator must have identified the request parameters as matching a configured tile cache.
If you are using a TIME parameter in your request, the only way the GWC mediator said it does match a tile cache is if you have a parameter filter[1] for the TIME parameter keyword configured for the layer.
Check GWC.isCachingPossible(TileLayer, GetMapRequest):

if (null != request.getTime() && !request.getTime().isEmpty()) {
if (!filterApplies(filters, request, “TIME”, requestMistmatchTarget)) {
return false;
}
}

If that’s not working, then report the bug back here please.

Now, at the current state of the gwc integration, you can’t configure a parameter filter for an embedded geoserver layer. What you can do is to configure a wmsLayer in /gwc/geowebcache.xml, name it something different than the geoserver layer (eg, “states” instead of “topp:states”), make it point to the geoserver layer you want to cache, and configure a parameter filter for time (this is about to change in the very near future. You’ll be able to configure these and other caching properties through the geoserver UI, and enabling parameter filters for STYLES, TIME, and ELEVATION will be as simple as a checkbox).

As Andrea mentioned, building a tile cache for all your TIME values might lead to a huge cache. Although you don’t need to pre-seed them all. You can rather let the cache being built on demand, and set up the disk quota limit so that gwc cleans up less recently/frequently used tiles up to a disk quota usage limit.

Hope that helps,
Gabriel

[1] <http://geowebcache.org/docs/current/configuration/layers/parameterfilters.html>

On Sun, Feb 12, 2012 at 2:25 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Feb 12, 2012 at 5:39 PM, John Armstrong <jrja@anonymised.com> wrote:

Hi, everybody.

I’ve got something hacked together so that my CoverageReader can handle
TIME parameters, and I’ve tested it with wms calls through curl. Now
I’m running into trouble when I try to hook it up to OpenLayers. It
seems that GeoWebCache takes the incoming request and assembles a “fake”
request from it with a fixed list of parameters. In
GeoServerTileLayer.buildGetMap():

params.put(“SERVICE”, “WMS”);
params.put(“VERSION”, “1.1.1”);
params.put(“REQUEST”, “GetMap”);
params.put(“LAYERS”, getName());
params.put(“SRS”, srs);
params.put(“FORMAT”, format);
params.put(“WIDTH”, String.valueOf(width));
params.put(“HEIGHT”, String.valueOf(height));
params.put(“BBOX”, bbox.toString());

params.put(“EXCEPTIONS”, GetMapRequest.SE_XML);
params.put(“STYLES”, “”);
params.put(“TRANSPARENT”, “true”);
params.put(GWC_SEED_INTERCEPT_TOKEN, “true”);

Map<String, String> filteredParams = tile.getFullParameters();
if (filteredParams.isEmpty()) {
filteredParams = getDefaultParameterFilters();
}
params.putAll(filteredParams);

the last line could, I suppose, be used to add other parameters to the
fake request, but tracing through the history of the tile object, I
can’t find any place where it might get added to tile.fullParameters.

Any ideas? How does this work for ImageMosaics? Thanks in advance.

It most likely doesn’t.

We actually did not check it, time + elevation would
require creating a separate tile cache for each different value passed in, which would
end up requiring a massive amount of disk space, so it’s a case I did not bother
to check when implementing time/elevation support.

Feel free to provide patches to make this case work (I suppose you have very
few values for time/elevation?)

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

5

On Sun, Feb 12, 2012 at 6:29 PM, John Armstrong <jrja@anonymised.com> wrote:

On 02/12/2012 12:25 PM, Andrea Aime wrote:

On Sun, Feb 12, 2012 at 5:39 PM, John Armstrong <jrja@anonymised.com> wrote:

Any ideas? How does this work for ImageMosaics? Thanks in advance.

It most likely doesn’t.

Feel free to provide patches to make this case work (I suppose you have very
few values for time/elevation?)

I’ll see if I can come up with something; we’re looking at updating a layer daily and being able to retrieve timestamped tiles from a BigTable-like datastore. I don’t really know about typical use cases, but I wouldn’t be surprised if a user wanted to scroll through, say, a month’s worth of updates at various zoom levels. It all depends on what you mean by “very few”.

Tens? Or else, the area is very small (county or smaller), or the zoom level is not thought to match Google Maps (does not go down to 20).
See this spreadsheet to estimate the size of a tile cache for a fully developed tile cache, on disk,
for a certain area:
https://docs.google.com/spreadsheet/ccc?key=0Aq3GF1EnUyHEdHlDSW03cno4NzUzRFVHQzlGeU9YYXc#gid=0

I’ve filled it with the Massachusetts bounds, at level 20 the tile cache uses 4TB of disk.
You don’t go very far with one of the above for each possible time/elevation value…

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

6

On 02/12/2012 12:40 PM, Gabriel Roldan wrote:

If you are using a TIME parameter in your request, the only way the GWC mediator said it does match a tile cache is if you have a parameter filter[1] for the TIME parameter keyword configured for the layer.

Okay, this makes sense with the code I'm seeing, and I've been trying to figure out how to set such a parameter filter on one of my layers.

Check GWC.isCachingPossible(TileLayer, GetMapRequest):

    if (null != request.getTime() && !request.getTime().isEmpty()) {
        if (!filterApplies(filters, request, "TIME", requestMistmatchTarget)) {
            return false;
        }
    }

If that's not working, then report the bug back here please.

I don't see this method; our current geoserver is version 2.1.2, so I've been going off of its code. Is this new since then?

Now, at the current state of the gwc integration, you can't configure a parameter filter for an embedded geoserver layer. What you can do is to configure a wmsLayer in <data dir>/gwc/geowebcache.xml, name it something different than the geoserver layer (eg, "states" instead of "topp:states"), make it point to the geoserver layer you want to cache, and configure a parameter filter for time (this is about to change in the very near future. You'll be able to configure these and other caching properties through the geoserver UI, and enabling parameter filters for STYLES, TIME, and ELEVATION will be as simple as a checkbox).

Okay, so to check if I've got this: I can't set this up directly on my own layer. What I have to do is set up a GWC layer that's backed by my own layer and which does take a TIME filter, and then point OpenLayers to that. Right?

As Andrea mentioned, building a tile cache for all your TIME values might lead to a huge cache. Although you don't need to pre-seed them all. You can rather let the cache being built on demand, and set up the disk quota limit so that gwc cleans up less recently/frequently used tiles up to a disk quota usage limit.

Oh, yes, we're not pre-seeding the whole thing. We just want caching to speed up repeated access to recently-viewed tiles.

Thanks again for your help. If I've understood you I think I can make this work.

7

On Sun, Feb 12, 2012 at 2:49 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Feb 12, 2012 at 6:29 PM, John Armstrong <jrja@anonymised.com> wrote:

On 02/12/2012 12:25 PM, Andrea Aime wrote:

On Sun, Feb 12, 2012 at 5:39 PM, John Armstrong <jrja@anonymised.com> wrote:

Any ideas? How does this work for ImageMosaics? Thanks in advance.

It most likely doesn’t.

Feel free to provide patches to make this case work (I suppose you have very
few values for time/elevation?)

I’ll see if I can come up with something; we’re looking at updating a layer daily and being able to retrieve timestamped tiles from a BigTable-like datastore. I don’t really know about typical use cases, but I wouldn’t be surprised if a user wanted to scroll through, say, a month’s worth of updates at various zoom levels. It all depends on what you mean by “very few”.

Tens? Or else, the area is very small (county or smaller), or the zoom level is not thought to match Google Maps (does not go down to 20).
See this spreadsheet to estimate the size of a tile cache for a fully developed tile cache, on disk,
for a certain area:
https://docs.google.com/spreadsheet/ccc?key=0Aq3GF1EnUyHEdHlDSW03cno4NzUzRFVHQzlGeU9YYXc#gid=0

I’ve filled it with the Massachusetts bounds, at level 20 the tile cache uses 4TB of disk.

My read of it is it’s 4G?

You don’t go very far with one of the above for each possible time/elevation value…

Yet that’s still the danger wrt disk usage, and where disk quota and min/max cached zoom levels come to play [1].
Yeah, you won’t have exactly a static tile cache, but you need to make some compromises when disk usage is a concern.

Cheers,
Gabriel.

[1] Functionality already on stable branch. Documentation to be committed: <https://github.com/bmmpxf/geowebcache/commit/202ae13b6b8f8a44520093fca1f208c7b9fa8755>

Cheers

Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

8

On Sun, Feb 12, 2012 at 3:19 PM, John Armstrong <jrja@anonymised.com.> wrote:

On 02/12/2012 12:40 PM, Gabriel Roldan wrote:

If you are using a TIME parameter in your request, the only way the GWC mediator said it does match a tile cache is if you have a parameter filter[1] for the TIME parameter keyword configured for the layer.

Okay, this makes sense with the code I’m seeing, and I’ve been trying to figure out how to set such a parameter filter on one of my layers.

Check GWC.isCachingPossible(TileLayer, GetMapRequest):

if (null != request.getTime() && !request.getTime().isEmpty()) {
if (!filterApplies(filters, request, “TIME”, requestMistmatchTarget)) {
return false;
}
}

If that’s not working, then report the bug back here please.

I don’t see this method; our current geoserver is version 2.1.2, so I’ve been going off of its code. Is this new since then?

It looks like it is there in 2.1.2: <http://svn.codehaus.org/geoserver/tags/2.1.2/src/gwc/src/main/java/org/geoserver/gwc/GWC.java>

Now, at the current state of the gwc integration, you can’t configure a parameter filter for an embedded geoserver layer. What you can do is to configure a wmsLayer in /gwc/geowebcache.xml, name it something different than the geoserver layer (eg, “states” instead of “topp:states”), make it point to the geoserver layer you want to cache, and configure a parameter filter for time (this is about to change in the very near future. You’ll be able to configure these and other caching properties through the geoserver UI, and enabling parameter filters for STYLES, TIME, and ELEVATION will be as simple as a checkbox).

Okay, so to check if I’ve got this: I can’t set this up directly on my own layer. What I have to do is set up a GWC layer that’s backed by my own layer and which does take a TIME filter, and then point OpenLayers to that. Right?

Right.

As Andrea mentioned, building a tile cache for all your TIME values might lead to a huge cache. Although you don’t need to pre-seed them all. You can rather let the cache being built on demand, and set up the disk quota limit so that gwc cleans up less recently/frequently used tiles up to a disk quota usage limit.

Oh, yes, we’re not pre-seeding the whole thing. We just want caching to speed up repeated access to recently-viewed tiles.

See the other reply, you may want to combine with disk quota and min/max cached zoom levels (available on nightlies, or wait for the next release).

Cheers,
Gabriel

Thanks again for your help. If I’ve understood you I think I can make this work.

–
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

9

On Sun, Feb 12, 2012 at 7:54 PM, Gabriel Roldan <groldan@anonymised.com…> wrote:

Tens? Or else, the area is very small (county or smaller), or the zoom level is not thought to match Google Maps (does not go down to 20).
See this spreadsheet to estimate the size of a tile cache for a fully developed tile cache, on disk,
for a certain area:
https://docs.google.com/spreadsheet/ccc?key=0Aq3GF1EnUyHEdHlDSW03cno4NzUzRFVHQzlGeU9YYXc#gid=0

I’ve filled it with the Massachusetts bounds, at level 20 the tile cache uses 4TB of disk.

My read of it is it’s 4G?

Doh, right!

You don’t go very far with one of the above for each possible time/elevation value…

Yet that’s still the danger wrt disk usage, and where disk quota and min/max cached zoom levels come to play [1].
Yeah, you won’t have exactly a static tile cache, but you need to make some compromises when disk usage is a concern.

Hmm… so far I’ve hit few cases where the disk usage is not a concern, more often than not the area
that one needs to depict seems to grow faster than the ability to buy disk for it (meaning, local data
handlers often cannot afford a serious NAS, and those that do often have continents to deal with,
at that point the NAS are small anyways… and disk quota does not cluster, making GWC not usable in those cases
since HA is a requirement … )

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

10

On 02/12/2012 02:56 PM, Andrea Aime wrote:

On Sun, Feb 12, 2012 at 7:54 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

Yet that’s still the danger wrt disk usage, and where disk quota and min/max cached zoom levels come to play [1].
Yeah, you won’t have exactly a static tile cache, but you need to make some compromises when disk usage is a concern.

Hmm… so far I’ve hit few cases where the disk usage is not a concern, more often than not the area
that one needs to depict seems to grow faster than the ability to buy disk for it (meaning, local data
handlers often cannot afford a serious NAS, and those that do often have continents to deal with,
at that point the NAS are small anyways… and disk quota does not cluster, making GWC not usable in those cases
since HA is a requirement … )

Well, we’re not pre-seeding, and our pyramids only go down to… I think ZL14. Even those images are pretty small since the resolution of our data is only 35 geohash bits. And yes, I think we have disk quota set. Mostly the use of GWC is there to speed up reloading of recently-viewed layers within a short span of time.

11

On Sun, Feb 12, 2012 at 4:56 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Feb 12, 2012 at 7:54 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

Tens? Or else, the area is very small (county or smaller), or the zoom level is not thought to match Google Maps (does not go down to 20).
See this spreadsheet to estimate the size of a tile cache for a fully developed tile cache, on disk,
for a certain area:
https://docs.google.com/spreadsheet/ccc?key=0Aq3GF1EnUyHEdHlDSW03cno4NzUzRFVHQzlGeU9YYXc#gid=0

I’ve filled it with the Massachusetts bounds, at level 20 the tile cache uses 4TB of disk.

My read of it is it’s 4G?

Doh, right!

You don’t go very far with one of the above for each possible time/elevation value…

Yet that’s still the danger wrt disk usage, and where disk quota and min/max cached zoom levels come to play [1].
Yeah, you won’t have exactly a static tile cache, but you need to make some compromises when disk usage is a concern.

Hmm… so far I’ve hit few cases where the disk usage is not a concern, more often than not the area
that one needs to depict seems to grow faster than the ability to buy disk for it (meaning, local data
handlers often cannot afford a serious NAS, and those that do often have continents to deal with,
at that point the NAS are small anyways… and disk quota does not cluster, making GWC not usable in those cases
since HA is a requirement … )

Indeed. Working hard to get there.
Cheers,
Gabriel

Cheers

Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.