[Geoserver-devel] gwc configuration startup in separate thread

Hi all,

Currently the gwc module does its configuration in a separate thread. This is problematic for dbconfig because during that configuration gwc must access the catalog. The way session management is set up currently is that it is thread local based. So this leads to issues.

Describing the issue a bit more… the way the hibernate dao is setup is that its proxy will allow methods to be called on the catalog and create a session on demand… howver that session is closed after the method called. Which prevents any sort of lazy initialization. So what we do on startup is create a thread local session and keep it around until geoserver startup is complete. And so if gwc did its config in the main startup thread there would be no issue… but since it does not … kaboom.

So… thoughts on this issue? It would be easy enough to patch the gwc TileLayerDispatcher to load in the current thread… but i don’t know enough about gwc to know if that is a good idea.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

It’s a leftover from the days when GWC first needed GeoServer’s WMS to be ready to serve a getcapabilities document before GWC could initialize.

In theory it is still useful if you wanted to use GWC integrated in GeoServer and connect to another instance running in the same servlet container. I don’t think very many people need that, and it was very painful to write in the first place, so I’d be happy to get rid of it.

Are you trying to fix 2.1 or 2.0.x as well? Standalone GWC still needs the code for as long as people use the getcapabilities configuration, I (or Gabriel) would have to look into whether it can be toggled with a flag. In the long run I want to get rid of that kind of dynamic configuration anyway.

-Arne

On 10/4/10 6:50 PM, Justin Deoliveira wrote:

Hi all,

Currently the gwc module does its configuration in a separate thread. This is problematic for dbconfig because during that configuration gwc must access the catalog. The way session management is set up currently is that it is thread local based. So this leads to issues.

Describing the issue a bit more… the way the hibernate dao is setup is that its proxy will allow methods to be called on the catalog and create a session on demand… howver that session is closed after the method called. Which prevents any sort of lazy initialization. So what we do on startup is create a thread local session and keep it around until geoserver startup is complete. And so if gwc did its config in the main startup thread there would be no issue… but since it does not … kaboom.

So… thoughts on this issue? It would be easy enough to patch the gwc TileLayerDispatcher to load in the current thread… but i don’t know enough about gwc to know if that is a good idea.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Thanks for the explanation Arne. Makes more sense now.

On Mon, Oct 4, 2010 at 11:17 AM, Arne Kepp <arne@anonymised.com> wrote:

It’s a leftover from the days when GWC first needed GeoServer’s WMS to be ready to serve a getcapabilities document before GWC could initialize.

In theory it is still useful if you wanted to use GWC integrated in GeoServer and connect to another instance running in the same servlet container. I don’t think very many people need that, and it was very painful to write in the first place, so I’d be happy to get rid of it.

Are you trying to fix 2.1 or 2.0.x as well? Standalone GWC still needs the code for as long as people use the getcapabilities configuration, I (or Gabriel) would have to look into whether it can be toggled with a flag. In the long run I want to get rid of that kind of dynamic configuration anyway.

This would only be 2.1. On 2.1 the default is the CatalogConfiguration right? Can one set it to be the wms configuration on 2.1.x? For what it is worth here is a patch i had success with. Although i did not test gwc thoroughly after ward.

Index: src/main/java/org/geowebcache/layer/TileLayerDispatcher.java

— src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (revision 1188)
+++ src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (working copy)
@@ -64,14 +64,30 @@

public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
int loadDelay) {

  • this(gridSetBroker, configs, true, loadDelay);
  • }
  • public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
  • boolean threaded, int loadDelay) {

this.gridSetBroker = gridSetBroker;

this.configs = configs;

  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • if (threaded) {
  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • }
  • else {
  • try {
  • configuredLayers = new ConfigurationLoader(this, loadDelay).call();
  • }
  • catch (Exception e) {
  • throw new RuntimeException(e);
  • }
  • }
    }

public TileLayer getTileLayer(String layerIdent) throws GeoWebCacheException {

-Arne

On 10/4/10 6:50 PM, Justin Deoliveira wrote:

Hi all,

Currently the gwc module does its configuration in a separate thread. This is problematic for dbconfig because during that configuration gwc must access the catalog. The way session management is set up currently is that it is thread local based. So this leads to issues.

Describing the issue a bit more… the way the hibernate dao is setup is that its proxy will allow methods to be called on the catalog and create a session on demand… howver that session is closed after the method called. Which prevents any sort of lazy initialization. So what we do on startup is create a thread local session and keep it around until geoserver startup is complete. And so if gwc did its config in the main startup thread there would be no issue… but since it does not … kaboom.

So… thoughts on this issue? It would be easy enough to patch the gwc TileLayerDispatcher to load in the current thread… but i don’t know enough about gwc to know if that is a good idea.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Arne explained it well, for some reason you may want the initialization
to happen after some time to give sibling webapps a chance to load
before hitting them, or even non sibling webapps but say other apps
being started up as operating system daemons (say there's an init script
for both gwc and apache, and apache loads after tomcat, and gwc uses a
MapServer in that apache instance).

But yeah, in theory is only applies to GetCapabilitiesConfiguration that
we should kill, and the startup problem should be handled somehow else,
if at all, because I don't think gwc will try to hit the WMS only by
loading the XMLConfiguration.

So I would rather move the loadDelay as a property of the
GetCapabilitiesConfiguration object and let it sleep if it needs to
(like in, only if it's the first time it's loading the TileLayers)

Makes sense?
if so, Justin please feel free to keep using your patch until I come up
with a better fix for this situation.

Gabriel

On Mon, 2010-10-04 at 11:41 -0600, Justin Deoliveira wrote:

Thanks for the explanation Arne. Makes more sense now.

On Mon, Oct 4, 2010 at 11:17 AM, Arne Kepp <arne@anonymised.com>
wrote:
        It's a leftover from the days when GWC first needed
        GeoServer's WMS to be ready to serve a getcapabilities
        document before GWC could initialize.
        
        In theory it is still useful if you wanted to use GWC
        integrated in GeoServer and connect to another instance
        running in the same servlet container. I don't think very many
        people need that, and it was very painful to write in the
        first place, so I'd be happy to get rid of it.
        
        Are you trying to fix 2.1 or 2.0.x as well? Standalone GWC
        still needs the code for as long as people use the
        getcapabilities configuration, I (or Gabriel) would have to
        look into whether it can be toggled with a flag. In the long
        run I want to get rid of that kind of dynamic configuration
        anyway.
        
This would only be 2.1. On 2.1 the default is the CatalogConfiguration
right? Can one set it to be the wms configuration on 2.1.x? For what
it is worth here is a patch i had success with. Although i did not
test gwc thoroughly after ward.

Index: src/main/java/org/geowebcache/layer/TileLayerDispatcher.java

--- src/main/java/org/geowebcache/layer/TileLayerDispatcher.java
(revision 1188)
+++ src/main/java/org/geowebcache/layer/TileLayerDispatcher.java
(working copy)
@@ -64,14 +64,30 @@

     public TileLayerDispatcher(GridSetBroker gridSetBroker,
List<Configuration> configs,
             int loadDelay) {
+ this(gridSetBroker, configs, true, loadDelay);
+ }
+
+ public TileLayerDispatcher(GridSetBroker gridSetBroker,
List<Configuration> configs,
+ boolean threaded, int loadDelay) {
+
         this.gridSetBroker = gridSetBroker;

         this.configs = configs;

- ThreadFactory tfac = new CustomizableThreadFactory("GWC
Configuration loader thread");
- configLoadService = Executors.newSingleThreadExecutor(tfac);
- ConfigurationLoader loader = new ConfigurationLoader(this,
loadDelay);
- configurationLoadTask = configLoadService.submit(loader);
+ if (threaded) {
+ ThreadFactory tfac = new CustomizableThreadFactory("GWC
Configuration loader thread");
+ configLoadService =
Executors.newSingleThreadExecutor(tfac);
+ ConfigurationLoader loader = new
ConfigurationLoader(this, loadDelay);
+ configurationLoadTask = configLoadService.submit(loader);
+ }
+ else {
+ try {
+ configuredLayers = new ConfigurationLoader(this,
loadDelay).call();
+ }
+ catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
     }

     public TileLayer getTileLayer(String layerIdent) throws
GeoWebCacheException {

        -Arne
        
        On 10/4/10 6:50 PM, Justin Deoliveira wrote:
        > Hi all,
        >
        >
        > Currently the gwc module does its configuration in a
        > separate thread. This is problematic for dbconfig because
        > during that configuration gwc must access the catalog. The
        > way session management is set up currently is that it is
        > thread local based. So this leads to issues.
        >
        >
        > Describing the issue a bit more... the way the hibernate dao
        > is setup is that its proxy will allow methods to be called
        > on the catalog and create a session on demand... howver that
        > session is closed after the method called. Which prevents
        > any sort of lazy initialization. So what we do on startup is
        > create a thread local session and keep it around until
        > geoserver startup is complete. And so if gwc did its config
        > in the main startup thread there would be no issue... but
        > since it does not ... kaboom.
        >
        >
        > So... thoughts on this issue? It would be easy enough to
        > patch the gwc TileLayerDispatcher to load in the current
        > thread... but i don't know enough about gwc to know if that
        > is a good idea.
        >
        >
        > -Justin
        >
        > --
        > Justin Deoliveira
        > OpenGeo - http://opengeo.org
        > Enterprise support for open source geospatial.
        >
        >
        
--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

------------------------------------------------------------------------------
Virtualization is moving to the mainstream and overtaking non-virtualized
environment for deploying applications. Does it make network security
easier or more difficult to achieve? Read this whitepaper to separate the
two and get a better understanding.
http://p.sf.net/sfu/hp-phase2-d2d
_______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Gabriel Roldan
groldan@anonymised.com
Expert service straight from the developers

Looks fine to me too, alternatively could just use loadDelay < 1.

-Arne

On 10/4/10 7:41 PM, Justin Deoliveira wrote:

Thanks for the explanation Arne. Makes more sense now.

On Mon, Oct 4, 2010 at 11:17 AM, Arne Kepp <arne@anonymised.com> wrote:

It’s a leftover from the days when GWC first needed GeoServer’s WMS to be ready to serve a getcapabilities document before GWC could initialize.

In theory it is still useful if you wanted to use GWC integrated in GeoServer and connect to another instance running in the same servlet container. I don’t think very many people need that, and it was very painful to write in the first place, so I’d be happy to get rid of it.

Are you trying to fix 2.1 or 2.0.x as well? Standalone GWC still needs the code for as long as people use the getcapabilities configuration, I (or Gabriel) would have to look into whether it can be toggled with a flag. In the long run I want to get rid of that kind of dynamic configuration anyway.

This would only be 2.1. On 2.1 the default is the CatalogConfiguration right? Can one set it to be the wms configuration on 2.1.x? For what it is worth here is a patch i had success with. Although i did not test gwc thoroughly after ward.

Index: src/main/java/org/geowebcache/layer/TileLayerDispatcher.java

— src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (revision 1188)
+++ src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (working copy)
@@ -64,14 +64,30 @@

public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
int loadDelay) {

  • this(gridSetBroker, configs, true, loadDelay);
  • }
  • public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
  • boolean threaded, int loadDelay) {

this.gridSetBroker = gridSetBroker;

this.configs = configs;

  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • if (threaded) {
  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • }
  • else {
  • try {
  • configuredLayers = new ConfigurationLoader(this, loadDelay).call();
  • }
  • catch (Exception e) {
  • throw new RuntimeException(e);
  • }
  • }
    }

public TileLayer getTileLayer(String layerIdent) throws GeoWebCacheException {

-Arne

On 10/4/10 6:50 PM, Justin Deoliveira wrote:

Hi all,

Currently the gwc module does its configuration in a separate thread. This is problematic for dbconfig because during that configuration gwc must access the catalog. The way session management is set up currently is that it is thread local based. So this leads to issues.

Describing the issue a bit more… the way the hibernate dao is setup is that its proxy will allow methods to be called on the catalog and create a session on demand… howver that session is closed after the method called. Which prevents any sort of lazy initialization. So what we do on startup is create a thread local session and keep it around until geoserver startup is complete. And so if gwc did its config in the main startup thread there would be no issue… but since it does not … kaboom.

So… thoughts on this issue? It would be easy enough to patch the gwc TileLayerDispatcher to load in the current thread… but i don’t know enough about gwc to know if that is a good idea.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Sounds nicer, more concise :slight_smile:

So I guess i will stick with the patch locally until it gets fixed on gwc trunk and anyone trying out dbconfig in the meantime will have to disable gwc.

Thanks guys.

-Justin

On Mon, Oct 4, 2010 at 1:14 PM, Arne Kepp <arne@anonymised.com> wrote:

Looks fine to me too, alternatively could just use loadDelay < 1.

-Arne

On 10/4/10 7:41 PM, Justin Deoliveira wrote:

Thanks for the explanation Arne. Makes more sense now.

On Mon, Oct 4, 2010 at 11:17 AM, Arne Kepp <arne@anonymised.com> wrote:

It’s a leftover from the days when GWC first needed GeoServer’s WMS to be ready to serve a getcapabilities document before GWC could initialize.

In theory it is still useful if you wanted to use GWC integrated in GeoServer and connect to another instance running in the same servlet container. I don’t think very many people need that, and it was very painful to write in the first place, so I’d be happy to get rid of it.

Are you trying to fix 2.1 or 2.0.x as well? Standalone GWC still needs the code for as long as people use the getcapabilities configuration, I (or Gabriel) would have to look into whether it can be toggled with a flag. In the long run I want to get rid of that kind of dynamic configuration anyway.

This would only be 2.1. On 2.1 the default is the CatalogConfiguration right? Can one set it to be the wms configuration on 2.1.x? For what it is worth here is a patch i had success with. Although i did not test gwc thoroughly after ward.

Index: src/main/java/org/geowebcache/layer/TileLayerDispatcher.java

— src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (revision 1188)
+++ src/main/java/org/geowebcache/layer/TileLayerDispatcher.java (working copy)
@@ -64,14 +64,30 @@

public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
int loadDelay) {

  • this(gridSetBroker, configs, true, loadDelay);
  • }
  • public TileLayerDispatcher(GridSetBroker gridSetBroker, List configs,
  • boolean threaded, int loadDelay) {

this.gridSetBroker = gridSetBroker;

this.configs = configs;

  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • if (threaded) {
  • ThreadFactory tfac = new CustomizableThreadFactory(“GWC Configuration loader thread”);
  • configLoadService = Executors.newSingleThreadExecutor(tfac);
  • ConfigurationLoader loader = new ConfigurationLoader(this, loadDelay);
  • configurationLoadTask = configLoadService.submit(loader);
  • }
  • else {
  • try {
  • configuredLayers = new ConfigurationLoader(this, loadDelay).call();
  • }
  • catch (Exception e) {
  • throw new RuntimeException(e);
  • }
  • }
    }

public TileLayer getTileLayer(String layerIdent) throws GeoWebCacheException {

-Arne

On 10/4/10 6:50 PM, Justin Deoliveira wrote:

Hi all,

Currently the gwc module does its configuration in a separate thread. This is problematic for dbconfig because during that configuration gwc must access the catalog. The way session management is set up currently is that it is thread local based. So this leads to issues.

Describing the issue a bit more… the way the hibernate dao is setup is that its proxy will allow methods to be called on the catalog and create a session on demand… howver that session is closed after the method called. Which prevents any sort of lazy initialization. So what we do on startup is create a thread local session and keep it around until geoserver startup is complete. And so if gwc did its config in the main startup thread there would be no issue… but since it does not … kaboom.

So… thoughts on this issue? It would be easy enough to patch the gwc TileLayerDispatcher to load in the current thread… but i don’t know enough about gwc to know if that is a good idea.

-Justin


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.


Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.