[Geoserver-devel] GWC UI landed on trunk

GeoServer GeoServer Developer
1

Hi all,

this is a notification message to let you all know that the long
standing work going to support editing (GeoServer backed) tile layers
through the UI has landed on trunk.
It is also an encouragement request to try it out and report back any
issue. Being brand new code I don't expect it to be perfect.
On the bright side, the /gwc module test coverage has increased to 79
% (it was ~60, but lines of code went from 4400 to 7700) and the
/web/gwc module's went from 0% to 64% (as per cobertura:cobertura),
and lines of code from 901 to 4056.

This work (finally) closes GEOS-4460.
There is more to do yet based on early feedback though, which I'll be
creating specific jira issues for. But most of it will be moving stuff
around in the UI for a better, more consistent "caching" story.

So, what does it all mean?
Basically that for the embedded gwc users shouldn't need to edit
geowebcache.xml by hand anymore. They can create/edit gridsets and
tile layer propertied directly through the UI.

WARNING: It also comes with a backwards _incompatible_ change wrt
storage of tile layer information. As previously discused on this
mailing list[1], we decided it'd be better for the tile layer
configuration to be decoupled from the LayerInfo and LayerGroupInfo
metadata maps. So for the sake of simplicity and scalability,
geoserver tile layer configuration will be stored in a <data
directory>/gwc-layers directory, using one file per layer containing
the tile layer's XML representation using XStream. So at startup the
config will be migrated, whipping out stuff from LayerInfo and
LayerGroupInfo metadata maps and stored in /gwc-layers.

Finally, these changes should fix some of the top rated bug/issue
reports about integrated gwc:
- How do I define new gridsets: go to the UI and either create one
from scratch or use an existing one as template
- How do I _stop_ caching a given layer? Yes, seems basic, but truth
is since its inception, gwc integration assumed all geoserver layers
were to have a sibling cached layer. Now go to the layer edit page
"caching" tab or the layergroup edit page and uncheck the "create a
cached layer" checkbox.
- How do I add a grid subset to a geoserver tile layer, or remove one
of the pre-defined ones? Answer used to be "create a separate wmsLayer
in geowebcache.xml, and don't forget to name it differently". Now you
can just edit the layer gridsubsets through the UI.

That'd be about it. Please go ahead and give it a try. Eager of bug
reports or any other feedback here.

Cheers,
Gabriel

[1] <http://osgeo-org.1560.n6.nabble.com/asking-for-feedback-on-gwc-integration-improvements-td4335885.html&gt;
--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2

Hey Gabriel,

The ui is looking great, nice work. I took it for a spin today when doing some last testing for the layer/group and style per workspace changes, and unfortunately ran into issues testing the case we discussed earlier today when a layer group with the same name exists globally and within a workspace.

The DefaultTileLAyerCatalog does not like this. Which makes sense since it is totally unaware of layer group workspaces. However, something problematic that it does is that it allows me to create two such layer groups (although it does seem to fail in the background) but it prevents me from starting up the server. Essentially leaving the server configuration corrupt without a way to fix it past hacking on the configuration files directly.

Obviously we will have to fix this specific case but it brings me to a deeper issue that i meant to bring up previously. And that is that it might be wise to include an off switch all together for embedded gwc. I know I have run into problems with it myself, and others have reported similar issues on the list and the fix unfortunately seems to be often to simply remove the gwc jars altogether from a geoserver install. It would be nice to instead be able to simply set a system parameter or a global config property that would do this. I realize this is probably quite a bit of work though.

Anyways, I would like to work with you to get this specific issue resolved since as it stands this blocks the completion of GSIP 73.

On Mon, Mar 5, 2012 at 7:14 AM, Gabriel Roldan <groldan@anonymised.com> wrote:

Hi all,

this is a notification message to let you all know that the long
standing work going to support editing (GeoServer backed) tile layers
through the UI has landed on trunk.
It is also an encouragement request to try it out and report back any
issue. Being brand new code I don’t expect it to be perfect.
On the bright side, the /gwc module test coverage has increased to 79
% (it was ~60, but lines of code went from 4400 to 7700) and the
/web/gwc module’s went from 0% to 64% (as per cobertura:cobertura),
and lines of code from 901 to 4056.

This work (finally) closes GEOS-4460.
There is more to do yet based on early feedback though, which I’ll be
creating specific jira issues for. But most of it will be moving stuff
around in the UI for a better, more consistent “caching” story.

So, what does it all mean?
Basically that for the embedded gwc users shouldn’t need to edit
geowebcache.xml by hand anymore. They can create/edit gridsets and
tile layer propertied directly through the UI.

WARNING: It also comes with a backwards incompatible change wrt
storage of tile layer information. As previously discused on this
mailing list[1], we decided it’d be better for the tile layer
configuration to be decoupled from the LayerInfo and LayerGroupInfo
metadata maps. So for the sake of simplicity and scalability,
geoserver tile layer configuration will be stored in a /gwc-layers directory, using one file per layer containing
the tile layer’s XML representation using XStream. So at startup the
config will be migrated, whipping out stuff from LayerInfo and
LayerGroupInfo metadata maps and stored in /gwc-layers.

Finally, these changes should fix some of the top rated bug/issue
reports about integrated gwc:

  • How do I define new gridsets: go to the UI and either create one
    from scratch or use an existing one as template
  • How do I stop caching a given layer? Yes, seems basic, but truth
    is since its inception, gwc integration assumed all geoserver layers
    were to have a sibling cached layer. Now go to the layer edit page
    “caching” tab or the layergroup edit page and uncheck the “create a
    cached layer” checkbox.
  • How do I add a grid subset to a geoserver tile layer, or remove one
    of the pre-defined ones? Answer used to be “create a separate wmsLayer
    in geowebcache.xml, and don’t forget to name it differently”. Now you
    can just edit the layer gridsubsets through the UI.

That’d be about it. Please go ahead and give it a try. Eager of bug
reports or any other feedback here.

Cheers,
Gabriel

[1] <http://osgeo-org.1560.n6.nabble.com/asking-for-feedback-on-gwc-integration-improvements-td4335885.html>

Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

Hey Justin,

On Mon, Mar 5, 2012 at 10:47 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hey Gabriel,

The ui is looking great, nice work.

cool, thanks.

I took it for a spin today when doing
some last testing for the layer/group and style per workspace changes, and
unfortunately ran into issues testing the case we discussed earlier today
when a layer group with the same name exists globally and within a
workspace.

I don't remember having such discussion (yesterday) but don't trust my
mind anyway.

The DefaultTileLAyerCatalog does not like this. Which makes sense since it
is totally unaware of layer group workspaces. However, something problematic
that it does is that it allows me to create two such layer groups (although
it does seem to fail in the background) but it prevents me from starting up
the server. Essentially leaving the server configuration corrupt without a
way to fix it past hacking on the configuration files directly.

Okay. Yes it makes sense for it to fail as layer group names have been
always unique. Unfortunately GWC doesn't distinguish between layer
identity (name) and identifier (id). It uses name as both, which is
bad, and something I already started to change, although not sure when
will be able to make a full switch.
So far the integrated GWC layers use a unique identifier to match it's
corresponding layerinfo or layergroupinfo, which is a step forward in
the right direction.

Pulling up the following sentence so my comment doesn't get too far
away from the above:

Anyways, I would like to work with you to get this specific issue resolved
since as it stands this blocks the completion of GSIP 73.

Sounds good. For the time being, I just committed a patch that avoids
creating a tile layer for a layergroup with duplicate name. The log
will get an error message on purpose, but won't avoid the layergroup
to be saved.
With this you should be able to commit your patch for GSIP 73.
Next step is to find a way forward on integrating gwc with the changes
from GSIP 73, and perhaps with the whole local workspace work, in a
nicer way. For which I'll try to summarize a plan on a separate thread
taking into account what we've talked about today on Skype and my
research after it.

Aside: I found another gap that'd prevent proper functioning: it seems
it is possible to create two layer groups with the same name in the
same workspace. What I did is to create a separate "tasmania" layer
group in the "topp" workspace, save it, and after that assign it the
default workspace and save it again. It got saved and both showed up
in the layer group page.

Obviously we will have to fix this specific case

Done as far as it goes as noted previously. I tried your patch for
GSIP 73 without the commits for GEOS-4460 and noted that, although it
didn't break, it didn't work nicely with gwc either. Like in GSIP 73
doesn't take gwc into account at all. The tile layer settings do get
created as part of the metadata map for a layergroup that has a
workspace set, but that tile layer will never see the outer world,
just because gwc is not prepared to do so. In any case I take it as my
responsibility as I haven't had a chance to look deeper into what all
these local workspace related work means for integrated gwc. And we
just found out with duplicate layer groups the first or worse symptom
of it so far. That said, it is hard being the only one that cares and
that gets the complaints, or so I feel. In any case I plea (again) to
the wider geoserver developers community to think also on what the
consequences for gwc would be upon any proposal that might affect it,
and to provide patches or raise the discussion early so I can make
sure to ask for time to spend on it. Recognizing gwc is an important
part of geoserver, whether one uses it himself or not, or has the time
and interest to become a contributor or not.

but it brings me to a
deeper issue that i meant to bring up previously. And that is that it might
be wise to include an off switch all together for embedded gwc. I know I
have run into problems with it myself, and others have reported similar
issues on the list and the fix unfortunately seems to be often to simply
remove the gwc jars altogether from a geoserver install. It would be nice to
instead be able to simply set a system parameter or a global config property
that would do this. I realize this is probably quite a bit of work though.

I would rather work towards making it robust enough so that if you
don't want to use it just remove the tile layers and disable the
'create a tile layer for every new added layer' functionality. Both
things are already possible, as is disabling the specific gwc provided
services (tms, wmts, wms-c). Diskquota not breaking a clustered (well,
load balanced) set up is not yet, unfortunately. I think that's what
you meant.
A global enablement switch would be indeed quite some work, and it's
already quite hard to keep gwc completely decoupled. In fact I think
it's the only non-extension module that's pluggable.
Patches are welcomed though. Although a simpler way of achieving that
would be a separate web module that doesn't include the gwc
dependency, or rather a profile in the current web-app module that
excludes it, so that you can have two war files, one with gwc and
another without.

Cheers,
Gabriel

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

4

Hey Gabriel, thanks for looking at this and fixing. The issue with two layer groups with the same name in the same workspace is fixed now, silly mistake by me pushing in a bade change to the last commit on that branch.

On Tue, Mar 6, 2012 at 7:24 PM, Gabriel Roldan <groldan@anonymised.com501…> wrote:

Hey Justin,

On Mon, Mar 5, 2012 at 10:47 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hey Gabriel,

The ui is looking great, nice work.

cool, thanks.

I took it for a spin today when doing
some last testing for the layer/group and style per workspace changes, and
unfortunately ran into issues testing the case we discussed earlier today
when a layer group with the same name exists globally and within a
workspace.

I don’t remember having such discussion (yesterday) but don’t trust my
mind anyway.

The DefaultTileLAyerCatalog does not like this. Which makes sense since it
is totally unaware of layer group workspaces. However, something problematic
that it does is that it allows me to create two such layer groups (although
it does seem to fail in the background) but it prevents me from starting up
the server. Essentially leaving the server configuration corrupt without a
way to fix it past hacking on the configuration files directly.

Okay. Yes it makes sense for it to fail as layer group names have been
always unique. Unfortunately GWC doesn’t distinguish between layer
identity (name) and identifier (id). It uses name as both, which is
bad, and something I already started to change, although not sure when
will be able to make a full switch.
So far the integrated GWC layers use a unique identifier to match it’s
corresponding layerinfo or layergroupinfo, which is a step forward in
the right direction.

Pulling up the following sentence so my comment doesn’t get too far
away from the above:

Anyways, I would like to work with you to get this specific issue resolved
since as it stands this blocks the completion of GSIP 73.

Sounds good. For the time being, I just committed a patch that avoids
creating a tile layer for a layergroup with duplicate name. The log
will get an error message on purpose, but won’t avoid the layergroup
to be saved.
With this you should be able to commit your patch for GSIP 73.
Next step is to find a way forward on integrating gwc with the changes
from GSIP 73, and perhaps with the whole local workspace work, in a
nicer way. For which I’ll try to summarize a plan on a separate thread
taking into account what we’ve talked about today on Skype and my
research after it.

Aside: I found another gap that’d prevent proper functioning: it seems
it is possible to create two layer groups with the same name in the
same workspace. What I did is to create a separate “tasmania” layer
group in the “topp” workspace, save it, and after that assign it the
default workspace and save it again. It got saved and both showed up
in the layer group page.

Obviously we will have to fix this specific case

Done as far as it goes as noted previously. I tried your patch for
GSIP 73 without the commits for GEOS-4460 and noted that, although it
didn’t break, it didn’t work nicely with gwc either. Like in GSIP 73
doesn’t take gwc into account at all. The tile layer settings do get
created as part of the metadata map for a layergroup that has a
workspace set, but that tile layer will never see the outer world,
just because gwc is not prepared to do so. In any case I take it as my
responsibility as I haven’t had a chance to look deeper into what all
these local workspace related work means for integrated gwc. And we
just found out with duplicate layer groups the first or worse symptom
of it so far. That said, it is hard being the only one that cares and
that gets the complaints, or so I feel. In any case I plea (again) to
the wider geoserver developers community to think also on what the
consequences for gwc would be upon any proposal that might affect it,
and to provide patches or raise the discussion early so I can make
sure to ask for time to spend on it. Recognizing gwc is an important
part of geoserver, whether one uses it himself or not, or has the time
and interest to become a contributor or not.

but it brings me to a
deeper issue that i meant to bring up previously. And that is that it might
be wise to include an off switch all together for embedded gwc. I know I
have run into problems with it myself, and others have reported similar
issues on the list and the fix unfortunately seems to be often to simply
remove the gwc jars altogether from a geoserver install. It would be nice to
instead be able to simply set a system parameter or a global config property
that would do this. I realize this is probably quite a bit of work though.

I would rather work towards making it robust enough so that if you
don’t want to use it just remove the tile layers and disable the
‘create a tile layer for every new added layer’ functionality. Both
things are already possible, as is disabling the specific gwc provided
services (tms, wmts, wms-c). Diskquota not breaking a clustered (well,
load balanced) set up is not yet, unfortunately. I think that’s what
you meant.
A global enablement switch would be indeed quite some work, and it’s
already quite hard to keep gwc completely decoupled. In fact I think
it’s the only non-extension module that’s pluggable.
Patches are welcomed though. Although a simpler way of achieving that
would be a separate web module that doesn’t include the gwc
dependency, or rather a profile in the current web-app module that
excludes it, so that you can have two war files, one with gwc and
another without.

Cheers,
Gabriel

–
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

On Wed, Mar 7, 2012 at 3:24 AM, Gabriel Roldan <groldan@anonymised.com…> wrote:

A global enablement switch would be indeed quite some work, and it’s
already quite hard to keep gwc completely decoupled. In fact I think
it’s the only non-extension module that’s pluggable.

Hmm… you sure?. As far as I know, you can safely remove restconfig,
wcs 1.0 and wcs 1.1 (if you remove also the gui components) and quite
likely wms too, if you remove its GUI as well.
WFS is tricker, as there is a number of modules depending on it (wms in particular)

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

6

On Wed, Mar 7, 2012 at 3:24 AM, Gabriel Roldan <groldan@anonymised.com…> wrote:

I would rather work towards making it robust enough so that if you
don’t want to use it just remove the tile layers and disable the
‘create a tile layer for every new added layer’ functionality. Both
things are already possible, as is disabling the specific gwc provided
services (tms, wmts, wms-c). Diskquota not breaking a clustered (well,
load balanced) set up is not yet, unfortunately. I think that’s what
you meant.

If I can share some bits of experience, we have been removing integrated GWC
since the start of the GS 2.1.x series because it can make GeoServer unstable
in unpredictable ways, it’s easier to trigger the misbehaviors with “direct gwc
integration” enabled but eventually they happen anyways.

We could not find a way to reproduce them, but when they happen we have
seen high cpu load (but nobody asked for a seed) and/or OOM, eventually
making GeoServer unresponsive.
The times this happened we were supposed to be in pre-production already,
too late in the game for an investigation, so we just removed it and put it
external.

Which is a pity, since the integrated one has a significant speed advantage
when it comes to seeding

Sorry I can’t offer more specific details on how to make this happen, if I had
a recipe I would have opened a bug report months ago :frowning:

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

7

On Wed, Mar 7, 2012 at 5:26 AM, Andrea Aime
<andrea.aime@anonymised.com> wrote:

On Wed, Mar 7, 2012 at 3:24 AM, Gabriel Roldan <groldan@anonymised.com> wrote:

I would rather work towards making it robust enough so that if you
don't want to use it just remove the tile layers and disable the
'create a tile layer for every new added layer' functionality. Both
things are already possible, as is disabling the specific gwc provided
services (tms, wmts, wms-c). Diskquota not breaking a clustered (well,
load balanced) set up is not yet, unfortunately. I think that's what
you meant.

If I can share some bits of experience, we have been removing integrated GWC
since the start of the GS 2.1.x series because it can make GeoServer
unstable
in unpredictable ways, it's easier to trigger the misbehaviors with "direct
gwc
integration" enabled but eventually they happen anyways.

We could not find a way to reproduce them, but when they happen we have
seen high cpu load (but nobody asked for a seed) and/or OOM, eventually
making GeoServer unresponsive.
The times this happened we were supposed to be in pre-production already,
too late in the game for an investigation, so we just removed it and put it
external.

Which is a pity, since the integrated one has a significant speed advantage
when it comes to seeding

Sorry I can't offer more specific details on how to make this happen, if I
had
a recipe I would have opened a bug report months ago :frowning:

okay, thanks for the insight. So can we say it happens('ed) with
direct wms integration enabled and while processing GetMap requests
that a) do match a tile, b) don't, c) both?

I'll try to reproduce with a long living jmeter run but it'd be good
to know which if the cases causes it.

TIA,
Gabriel

8

On Wed, Mar 7, 2012 at 1:42 PM, Gabriel Roldan <groldan@anonymised.com…> wrote:

okay, thanks for the insight. So can we say it happens('ed) with

direct wms integration enabled and while processing GetMap requests
that a) do match a tile, b) don’t, c) both?

I’ll try to reproduce with a long living jmeter run but it’d be good
to know which if the cases causes it.

As said, if I knew something about it I would tell.
It happened with real world traffic, with random WMS requests, some aligned
to the grid, many not, direct wms integration seemed to make the problem
more likely to happen, disabling it it was happening less frequently, but
in the end we had to remove GWC altogheter.

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

9

On Wed, Mar 7, 2012 at 10:07 AM, Andrea Aime
<andrea.aime@anonymised.com> wrote:

On Wed, Mar 7, 2012 at 1:42 PM, Gabriel Roldan <groldan@anonymised.com> wrote:

okay, thanks for the insight. So can we say it happens('ed) with
direct wms integration enabled and while processing GetMap requests
that a) do match a tile, b) don't, c) both?

I'll try to reproduce with a long living jmeter run but it'd be good
to know which if the cases causes it.

As said, if I knew something about it I would tell.
It happened with real world traffic, with random WMS requests, some aligned
to the grid, many not, direct wms integration seemed to make the problem
more likely to happen, disabling it it was happening less frequently, but
in the end we had to remove GWC altogheter.

due to OOM? if don't remember don't worry.

Gabriel