[Geoserver-devel] locking & TypeRepository

GeoServer GeoServer Developer
,
1

As far as I can tell from the source code, it seems that the locking of

features is completely implemented in java and not in the database.

Wouldn’t it be better, to implement locking in the database through the

data source? Especially for somebody that wants to use geoserver just as

an additional front-end to a common database. i.e. non-wfs clients and

wfs clients are accessing the data concurrently.

Yes, you are exactly correct. We are working on that functionality right

now. Most of the work is in geotools, we’ve done away with

TypeRepository. Most of the work is in the data module of geotools. The

locking functions are mostly in org.geotools.data.FeatureLocking. They

use a locking manager, with the default being an InProcessLockingManager,

which is like the TypeRepository, all java. The PostgisLockingManager

should be along soon, working with Postgis 0.8 when it comes out, as

Refractions added new locking backend locking methods. ArcSde and Oracle

will also be supported in time. It’s a bit trickier than it looks at

first, since the wfs notion of locking is a bit different than most db’s

notions.

I’ve just downloaded the latest nightly build from geotools and I’ve seen that some changes are going on in the area of locking. My point is just, the locking api should be flexible enough to work with an exisiting locking system, for instance in an existing gis system.

Another question concerning the TypeRepository.

org.vfny.geoserver.config.TypeRepository is a concrete class. It loads

the type information from info.xml and the corresponding xml schema from

schema.xml. This is a fairly specific setup. We would want to load this

information from the database. Shouldn’t there be an interface that

defines the basic methods every TypeRepository must implement and

implementing classes that define a custom TypeRepository? Just to remove

the dependency from a specific implementation…

Hmmm, that’s an interesting point. We’re redoing configuration right now.

We’ve got a lot of changes going on in the wms-merge, making room for the

new geotools data io apis. The replacement for TypeRepository is

org.geotools.config.CatalogConfig. The way the configuration will work is

that you define your connection parameters in catalog.xml, defining your

DataStores, and then each FeatureType will reference the its DataStore to

get its connection. The locking methods of TypeRepository move to

geotools LockingManagers. As for schema.xml, where we would eventually

like to go is have datasources automatically generate their schemas.

I would very much welcome this. FeatureTypes are also generated dynamically from table metadata, so it shouldn’t be a too big problem to implement this. I could do it quick and dirty, but that is obviously not very usefull…

They do this now, making geotools FeatureTypes, but we can not turn those

into XML Schemas yet. The geotools FeatureTypes also need some

improvement before we can do that. Jody Garnett is planning on spending

the next two months or so on configuration, and I think he also wants to

split things up a bit more. So what you should do is look at the new

configuration (which is still evolving), and come up with a concrete

proposal of how you’d like things split up.

I was just thinking, it would be nice to load the meta data (feature type name, title, …) from the database.

If possible you can code up a new set of interfaces, and we could easily

roll the changes in. Perhaps Jody has more thoughts on this issue? You

should probably collaborate with him and get his ideas on configuration,

and figure out a way to fit loading config files from other sources into

his plans.

I’ve just started looking into geoserver and geotools, so I’ve got to familiarize with the source first…

I’m working on adapting geoserver (especially the wfs part, because we have already a wms server) to an existing gis system. I don’t know if you’ve ever thought about this kind of usage of geoserver. I’m not interested to make fundamental changes to the source, just to adapt it to the existing system (must notably an existing database design). That’s why I’d really welcome it if geoserver and geotools would allow this as painless as possible.

hope this helps,

sure, it did.

2

> > As far as I can tell from the source code, it seems that the locking of
> > features is completely implemented in java and not in the database.
> > Wouldn't it be better, to implement locking in the database through the
> > data source? Especially for somebody that wants to use geoserver just as

> > an additional front-end to a common database. i.e. non-wfs clients and
> > wfs clients are accessing the data concurrently.
> Yes, you are exactly correct. We are working on that functionality right
> now. Most of the work is in geotools, we've done away with
> TypeRepository. Most of the work is in the data module of geotools. The
> locking functions are mostly in org.geotools.data.FeatureLocking. They
> use a locking manager, with the default being an InProcessLockingManager,
> which is like the TypeRepository, all java. The PostgisLockingManager
> should be along soon, working with Postgis 0.8 when it comes out, as
> Refractions added new locking backend locking methods. ArcSde and Oracle
> will also be supported in time. It's a bit trickier than it looks at
> first, since the wfs notion of locking is a bit different than most db's
> notions.
I've just downloaded the latest nightly build from geotools and I've seen
that some changes are going on in the area of locking. My point is just, the
locking api should be flexible enough to work with an exisiting locking
system, for instance in an existing gis system.

I think that it should. Correct me if I'm wrong Jody, but I think you
can just implement a new LockingManager that is specific to your system.
You'll probably then have to provide a wrapper to your DataStores
(something akin to org.vfny.geoserver.config.DEFQueryFeatureLocking, but
with DataStores instead of FeatureLocking). You could then hack the
wrapper into DataStoreConfig, or you could do it more cleanly with new
geotools datasources, extending the old but with an additional param for
their factories. Then you would just use those params for connection
params.

> We've got a lot of changes going on in the wms-merge, making room for the
> new geotools data io apis. The replacement for TypeRepository is
> org.geotools.config.CatalogConfig. The way the configuration will work is
> that you define your connection parameters in catalog.xml, defining your
> DataStores, and then each FeatureType will reference the its DataStore to
> get its connection. The locking methods of TypeRepository move to
> geotools LockingManagers. As for schema.xml, where we would eventually
> like to go is have datasources automatically generate their schemas.
I would very much welcome this. FeatureTypes are also generated dynamically
from table metadata, so it shouldn't be a too big problem to implement this.
I could do it quick and dirty, but that is obviously not very usefull...

Quick and dirty's a start, if you want to have a go at it we'd welcome it.
I just don't have the time right now. Before it reaches maturity we could
just give users the option to use one way or the other.

> They do this now, making geotools FeatureTypes, but we can not turn those
> into XML Schemas yet. The geotools FeatureTypes also need some
> improvement before we can do that. Jody Garnett is planning on spending
> the next two months or so on configuration, and I think he also wants to
> split things up a bit more. So what you should do is look at the new
> configuration (which is still evolving), and come up with a concrete
> proposal of how you'd like things split up.
I was just thinking, it would be nice to load the meta data (feature type
name, title, ...) from the database.

Yes, that would be nice. I think the future may make that possible. One
thing we're working right now is org.geotools.data.Catalog, in the data
module of geotools. It's still in a pretty early change, but I think I
would like to see FeatureTypeInfo, at first just the name, title, ect.,
and in time perhaps full metadata and a real catalog service, that we
could then integrate into geoserver. If we did things right we could have
different implementations of Catalog, database backed, file backed, ect.

> If possible you can code up a new set of interfaces, and we could easily
> roll the changes in. Perhaps Jody has more thoughts on this issue? You
> should probably collaborate with him and get his ideas on configuration,
> and figure out a way to fit loading config files from other sources into
> his plans.
I've just started looking into geoserver and geotools, so I've got to
familiarize with the source first...

Yes, I definitely understand. It takes awhile to get a grasp on it all.
It sounds like you've got the basics though, and know what you want out of
it, which is always helpful.

I'm working on adapting geoserver (especially the wfs part, because we have
already a wms server) to an existing gis system. I don't know if you've ever
thought about this kind of usage of geoserver. I'm not interested to make
fundamental changes to the source, just to adapt it to the existing system
(must notably an existing database design). That's why I'd really welcome it
if geoserver and geotools would allow this as painless as possible.

I actually have never thought about this kind of usage of geoserver, but
yes, it does make sense. Since that's never been a design goal it won't
be as painless as possible, but if you're willing to get a little dirty
with the source code we're happy to help out. The primary goal of
geoserver is more to provide a very easy to use solution to allow people
to share geographic data. Something that's easy to set up and get running
even for less advanced users. Which is not at all mutually exclusive to
being integrated in existing gis systems, it's just the reason why I've
never thought about it before. I don't think it should be _too_ difficult
integrating with another gis system, and would certainly be easier than
starting a comparable wfs from scratch. But since we haven't designed
specifically for that it will take a bit of work.

    Chris

3

Chris Holmes wrote:

that some changes are going on in the area of locking. My point is just, the
locking api should be flexible enough to work with an exisiting locking
system, for instance in an existing gis system.
   

I think that it should. Correct me if I'm wrong Jody, but I think you can just implement a new LockingManager that is specific to your system. You'll probably then have to provide a wrapper to your DataStores (something akin to org.vfny.geoserver.config.DEFQueryFeatureLocking, but with DataStores instead of FeatureLocking). You could then hack the wrapper into DataStoreConfig, or you could do it more cleanly with new geotools datasources, extending the old but with an additional param for their factories. Then you would just use those params for connection params.

That is a good idea Chris, My focus has been on using DataStore provided locking (whatever it is).
I have not been greatly trouble by making the process configurable. But it seems this is needed:
- Postgis will only provide locking a the most recent release
- Situations like the above where external locking systems exist

Interesting question. What is the existing locking system?

Jody

4

I would very much welcome this. FeatureTypes are also generated
dynamically from table metadata, so it shouldn't be a too big problem

to implement this. I could do it quick and dirty, but that is
obviously not very usefull...

Quick and dirty's a start, if you want to have a go at it we'd welcome

it.

I just don't have the time right now. Before it reaches maturity we

could

just give users the option to use one way or the other.

OK, I can have a go at it. Were would be the best place to add it?
Should I create a new class (for instance XmlSchemaGenerator) that takes
a FeatureType as input and outputs an xml schema? Or should this be
added to FeatureType directly (add method getXmlSchema())?
Probably a configuration option must be added that allows to specify
whether to load xml schemas from disk or to use this new mechanism.

One thing to note. I'm just interested in 'simple' features. The
features in my case are stored in one db table, so there is no nesting
of attributes.

Any suggestions, ideas, ...?

Simon Räss

5

Simon Räss wrote:

to implement this. I could do it quick and dirty, but that is

obviously not very usefull...
     

Quick and dirty's a start, if you want to have a go at it we'd welcome
   

it.

I just don't have the time right now. Before it reaches maturity we
   

could

just give users the option to use one way or the other.
   
OK, I can have a go at it. Were would be the best place to add it?
Should I create a new class (for instance XmlSchemaGenerator) that takes
a FeatureType as input and outputs an xml schema? Or should this be
added to FeatureType directly (add method getXmlSchema())?
Probably a configuration option must be added that allows to specify
whether to load xml schemas from disk or to use this new mechanism.

I have also wished to see the amount of configuration required be reduced, generating based on FeatureType is a wonderful idea.
I have tried to start a Catalog api in geotools2 that would hold the kind of information required. I have not managed to port FeatureTypeInfo over yet - but it is certaintly where I would start.

One thing to note. I'm just interested in 'simple' features. The
features in my case are stored in one db table, so there is no nesting
of attributes.

Any suggestions, ideas, ...?

I would start by comparing the information available in FeatureTypeInfo to that available through FeatureType.

There is a ISO specification that talks about Catalog requirements. In the short term I am more interested in having enough functionality for GeoServer then meeting a specification.

Cheers,

Jody

6

>> I would very much welcome this. FeatureTypes are also generated
>> dynamically from table metadata, so it shouldn't be a too big problem

>> to implement this. I could do it quick and dirty, but that is
>> obviously not very usefull...
> Quick and dirty's a start, if you want to have a go at it we'd welcome
it.
> I just don't have the time right now. Before it reaches maturity we
could
> just give users the option to use one way or the other.

OK, I can have a go at it. Were would be the best place to add it?
Should I create a new class (for instance XmlSchemaGenerator) that takes
a FeatureType as input and outputs an xml schema? Or should this be
added to FeatureType directly (add method getXmlSchema())?

No, don't add to the FeatureType directly. GML is just a format, so we
wouldn't want to support FeatureType.toShape() and such, instead it's done
in the specific data modules. I think the best thing to do would be to
put it in org.geotools.gml.producer, following FeatureTransformer, which
is used to turn Features into GML. If possible try to extend
org.geotools.xml.TranformerBase in the resources module, using SAX
generation methods. If you want to do something quicker and dirtier than
that we can just put the class in geoserver. Either way, you don't have
commit rights to either, so just write it up in the
org.geotools.gml.producer package. Then you can submit it for code review
and you'll be on your way to commit rights.

Probably a configuration option must be added that allows to specify
whether to load xml schemas from disk or to use this new mechanism.

Yes. I can handle that part for you if you'd like. I add config options
all the time, and it's easy once you know how.

One thing to note. I'm just interested in 'simple' features. The
features in my case are stored in one db table, so there is no nesting
of attributes.

That's fine. None of the geotools datasources yet support complex
features, so there's not much reason to do a bunch of work for complex
features.

Any suggestions, ideas, ...?

IanS, jump in here if you have any suggestions/corrections. Generating
gml shouldn't be too hard right now, because the Geotools feature model is
still not refined enough to handle all the intricacies of gml. We'd like
new gml readers (including a gml schema reader) to drive the featureType
model some more. For now most all the classes you need to support are in
DefaultAttributeType as inner classes. We've been moving to move that
hierarchy out into public classes, but haven't gotten around to it yet.

The other thing you'll have to make sure to do is leave hooks to put in
the target namespace and schema locations.

  Chris

7

I would very much welcome this. FeatureTypes are also generated
dynamically from table metadata, so it shouldn't be a too big problem

to implement this. I could do it quick and dirty, but that is
obviously not very usefull...

Quick and dirty's a start, if you want to have a go at it we'd welcome

it.

I just don't have the time right now. Before it reaches maturity we

could

just give users the option to use one way or the other.

OK, I can have a go at it. Were would be the best place to add it?
Should I create a new class (for instance XmlSchemaGenerator) that takes
a FeatureType as input and outputs an xml schema? Or should this be
added to FeatureType directly (add method getXmlSchema())?

No, don't add to the FeatureType directly. GML is just a format, so we
wouldn't want to support FeatureType.toShape() and such, instead it's done
in the specific data modules. I think the best thing to do would be to
put it in org.geotools.gml.producer, following FeatureTransformer, which
is used to turn Features into GML. If possible try to extend
org.geotools.xml.TranformerBase in the resources module, using SAX
generation methods. If you want to do something quicker and dirtier than
that we can just put the class in geoserver. Either way, you don't have
commit rights to either, so just write it up in the
org.geotools.gml.producer package. Then you can submit it for code review
and you'll be on your way to commit rights.

It soon came to my mind that FeatureType is the wrong place to add it. I will have a more thorough look at the classes you've mentioned and see which way I go. Thanks for your hints.

Probably a configuration option must be added that allows to specify
whether to load xml schemas from disk or to use this new mechanism.

Yes. I can handle that part for you if you'd like. I add config options
all the time, and it's easy once you know how.

I will gladly accept your offer as soon as I have a usable version.

One thing to note. I'm just interested in 'simple' features. The
features in my case are stored in one db table, so there is no nesting
of attributes.

That's fine. None of the geotools datasources yet support complex
features, so there's not much reason to do a bunch of work for complex
features.

Any suggestions, ideas, ...?

IanS, jump in here if you have any suggestions/corrections. Generating
gml shouldn't be too hard right now, because the Geotools feature model is
still not refined enough to handle all the intricacies of gml. We'd like
new gml readers (including a gml schema reader) to drive the featureType
model some more. For now most all the classes you need to support are in
DefaultAttributeType as inner classes. We've been moving to move that
hierarchy out into public classes, but haven't gotten around to it yet.

The other thing you'll have to make sure to do is leave hooks to put in
the target namespace and schema locations.

Try to do it as best as I can. I'm still new to all this (maybe I've got to re-read the specs...).