[Geoserver-devel] some resource pool refactoring

GeoServer GeoServer Developer
1

Hey folks,

I wanted to run some proposed refactoring of the ResourcePool class past everyone. Basically my goal is two-fold. The first and primary goal is to extend to the jdbcconfig module to allow for storing SLD contents along with configured style objects. It is really the last step to make the module able to store all catalog contents in the database.

The idea would be to make it possible to plug in a custom ResourcePool implementation via spring. The lookup mechanism is similar to how others work in that first we would look for an instance in the spring context, and if not found fall back to the default implementation.

A secondary goal is to allow for implementations to control how the various caches in ResourcePool are created. One thing I have been wanting to experiment with is the idea of distributed caches that GeoServers in a cluster could potentially share. This secondary goal is not an immediate one but I figured since i was mucking around in the vicinity i would make these changes as well.

All in all the changes are pretty straight forward, and mostly a pure refactoring with no behavioural changes.

https://github.com/jdeolive/geoserver/commit/2dd214507cd764486707ed93711f380ff4db0f71

Thoughts or opinions? Do people feel this is something proposal worthy? I should note that i was hoping to do this on 2.3.x.

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

2

Hi Justin

Please keep me informed about the distributed cache, we need such a cache for our authentication tokens.

Cheers
Christian

2013/1/15 Justin Deoliveira <jdeolive@anonymised.com…1501…>

Hey folks,

I wanted to run some proposed refactoring of the ResourcePool class past everyone. Basically my goal is two-fold. The first and primary goal is to extend to the jdbcconfig module to allow for storing SLD contents along with configured style objects. It is really the last step to make the module able to store all catalog contents in the database.

The idea would be to make it possible to plug in a custom ResourcePool implementation via spring. The lookup mechanism is similar to how others work in that first we would look for an instance in the spring context, and if not found fall back to the default implementation.

A secondary goal is to allow for implementations to control how the various caches in ResourcePool are created. One thing I have been wanting to experiment with is the idea of distributed caches that GeoServers in a cluster could potentially share. This secondary goal is not an immediate one but I figured since i was mucking around in the vicinity i would make these changes as well.

All in all the changes are pretty straight forward, and mostly a pure refactoring with no behavioural changes.

https://github.com/jdeolive/geoserver/commit/2dd214507cd764486707ed93711f380ff4db0f71

Thoughts or opinions? Do people feel this is something proposal worthy? I should note that i was hoping to do this on 2.3.x.

-Justin

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Master SQL Server Development, Administration, T-SQL, SSAS, SSIS, SSRS
and more. Get SQL Server skills now (including 2012) with LearnDevNow -
200+ hours of step-by-step video tutorials by Microsoft MVPs and experts.
SALE $99.99 this month only - learn more at:
http://p.sf.net/sfu/learnmore_122512

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

3

On Tue, Jan 15, 2013 at 11:28 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hey folks,

I wanted to run some proposed refactoring of the ResourcePool class past everyone. Basically my goal is two-fold. The first and primary goal is to extend to the jdbcconfig module to allow for storing SLD contents along with configured style objects. It is really the last step to make the module able to store all catalog contents in the database.

The idea would be to make it possible to plug in a custom ResourcePool implementation via spring. The lookup mechanism is similar to how others work in that first we would look for an instance in the spring context, and if not found fall back to the default implementation.

A secondary goal is to allow for implementations to control how the various caches in ResourcePool are created. One thing I have been wanting to experiment with is the idea of distributed caches that GeoServers in a cluster could potentially share. This secondary goal is not an immediate one but I figured since i was mucking around in the vicinity i would make these changes as well.

All in all the changes are pretty straight forward, and mostly a pure refactoring with no behavioural changes.

https://github.com/jdeolive/geoserver/commit/2dd214507cd764486707ed93711f380ff4db0f71

Thoughts or opinions? Do people feel this is something proposal worthy? I should note that i was hoping to do this on 2.3.x.

I’m good with the refactor allowing to change the resource pool, it is useful in general as people might want to
perform caching of resources their own way.

I agree that implementation wise this can be used to store styles somewhere else, but styles are not
the only thing that should be stored in the database if you want a centralized configuration.
How about freemarker templates, override xsd schemas, the security subsytem configuration,
or the configurations for the extension modules, the files in the www directory … and I’m
probably forgetting something else too.

It seems to me we’d need something different, like a ResourceFileAccessor of some kind, or
some extension to the catalog facade so that no direct access to the data directory
file system is ever made, possibly with events so that code can listen to direct changes
to the data dir contents.

Hmmm… and GWC won’t play good with that either, as the configuration code there
also assumes it can access a file system.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

4

Hey Andrea,

Agreed that storing all resources in a database or off the file system is a larger issue. For now my goal is just to focus on the catalog since it is typically the part of the configuration that changes most frequently, or is updated in an automated way.

That said, I have been using this problem to think about the larger goal of storing all resources in a database and making it so GeoServer can run without a file system. Here are some of my random thoughts.

The idea would be to use GeoServerResourceLoader as the base which I think makes sense since much of GeoServer uses this class for data directory / file system access. But obviously it works with java.io.File which is a problem. So the first step would be to make the api be stream based. There would be a lot of code around the edges that would port to the stream based methods but it could be done in a gradual way.

In terms of loading resources you could look at the current GeoServerResourceLoader api as a key value store where the keys are just file paths. Once the values are streams it should be possible to swap out the file based approach for one that stores blobs in a database . The key can remain the same:

stream = rl.findResource(“workspaces/topp/states/description.ftl”)

Whether that is a file in the data directory, or a row in a database becomes an implementation detail. Or at least that would be the idea.

Now gwc… well i don’t have any bright ideas about that :slight_smile: I guess the bits that live in the GeoServer extension could potentially be ported over to the new resource loader… but the upstream bits… not sure.

···

On Wed, Jan 16, 2013 at 3:18 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Tue, Jan 15, 2013 at 11:28 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hey folks,

I wanted to run some proposed refactoring of the ResourcePool class past everyone. Basically my goal is two-fold. The first and primary goal is to extend to the jdbcconfig module to allow for storing SLD contents along with configured style objects. It is really the last step to make the module able to store all catalog contents in the database.

The idea would be to make it possible to plug in a custom ResourcePool implementation via spring. The lookup mechanism is similar to how others work in that first we would look for an instance in the spring context, and if not found fall back to the default implementation.

A secondary goal is to allow for implementations to control how the various caches in ResourcePool are created. One thing I have been wanting to experiment with is the idea of distributed caches that GeoServers in a cluster could potentially share. This secondary goal is not an immediate one but I figured since i was mucking around in the vicinity i would make these changes as well.

All in all the changes are pretty straight forward, and mostly a pure refactoring with no behavioural changes.

https://github.com/jdeolive/geoserver/commit/2dd214507cd764486707ed93711f380ff4db0f71

Thoughts or opinions? Do people feel this is something proposal worthy? I should note that i was hoping to do this on 2.3.x.

I’m good with the refactor allowing to change the resource pool, it is useful in general as people might want to
perform caching of resources their own way.

I agree that implementation wise this can be used to store styles somewhere else, but styles are not
the only thing that should be stored in the database if you want a centralized configuration.
How about freemarker templates, override xsd schemas, the security subsytem configuration,
or the configurations for the extension modules, the files in the www directory … and I’m
probably forgetting something else too.

It seems to me we’d need something different, like a ResourceFileAccessor of some kind, or
some extension to the catalog facade so that no direct access to the data directory
file system is ever made, possibly with events so that code can listen to direct changes
to the data dir contents.

Hmmm… and GWC won’t play good with that either, as the configuration code there
also assumes it can access a file system.

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

On Wed, Jan 16, 2013 at 6:16 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Hey Andrea,

Agreed that storing all resources in a database or off the file system is a larger issue. For now my goal is just to focus on the catalog since it is typically the part of the configuration that changes most frequently, or is updated in an automated way.

That said, I have been using this problem to think about the larger goal of storing all resources in a database and making it so GeoServer can run without a file system. Here are some of my random thoughts.

The idea would be to use GeoServerResourceLoader as the base which I think makes sense since much of GeoServer uses this class for data directory / file system access. But obviously it works with java.io.File which is a problem. So the first step would be to make the api be stream based. There would be a lot of code around the edges that would port to the stream based methods but it could be done in a gradual way.

In terms of loading resources you could look at the current GeoServerResourceLoader api as a key value store where the keys are just file paths. Once the values are streams it should be possible to swap out the file based approach for one that stores blobs in a database . The key can remain the same:

stream = rl.findResource(“workspaces/topp/states/description.ftl”)

Whether that is a file in the data directory, or a row in a database becomes an implementation detail. Or at least that would be the idea.

Right, that’s what I was thinking as well.

Now gwc… well i don’t have any bright ideas about that :slight_smile: I guess the bits that live in the GeoServer extension could potentially be ported over to the new resource loader… but the upstream bits… not sure.

Eh, some parts of GWC will have to be refactored, several objects over there read directly from disk as they get initialized,
GWC also needs some notion of a facade to access the configuration bits, so that they also can be stored in places
other than a file system

Cheers
Andrea

–

==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it