[Geoserver-devel] virtual configurations, maps and profiles

Hi all,

I have been thinking about virtual configurations and resource
publishing split at the data structure level. Up until we have MapInfo:

MapInfo {

   String getName();

   List<LayerInfo> getLayers();

   List<StyleInfo> getStyles();

   List<String> getSRSs();

   ...
}

Which gives us basically the data side of the equation, but not the
services. One of the things undoubtedly people will want to do with a
virtual configuration is limit the services that can access a map. Now
we could do this:

MapInfo {

   List<ServiceInfo> getServices()

}

But this kind of seems off. Mainly because "map" lives on the catalog side of the fence, and services are a "configuration" entity. So the thought occurred to me to come up with another interface. For lack of a better name I am calling it a "profile". It would look something like:

ProfileInfo {

   String getName();

   List<Map> getMaps();

   List<ServiceInfo> getServices();

}

I think it would also make sense if we want to start attaching other bits of "virtual configuration". I am thinking things here like logging, contact info, etc....

Thoughts or comments?

-Justin

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Justin Deoliveira ha scritto:

Hi all,

I have been thinking about virtual configurations and resource
publishing split at the data structure level. Up until we have MapInfo:

MapInfo {

   String getName();

   List<LayerInfo> getLayers();

   List<StyleInfo> getStyles();

   List<String> getSRSs();

   ...
}

Not sure about styles and srs... SRS are part of the WMS configuration,
and styles WMS wise are simply not something that stands up on its own.
I guess by that you're trying to limit the styles that one can see
by doing direct access to them in styles=...?

Which gives us basically the data side of the equation, but not the
services. One of the things undoubtedly people will want to do with a
virtual configuration is limit the services that can access a map. Now
we could do this:

MapInfo {

   List<ServiceInfo> getServices()

}

But this kind of seems off. Mainly because "map" lives on the catalog side of the fence, and services are a "configuration" entity. So the thought occurred to me to come up with another interface. For lack of a better name I am calling it a "profile". It would look something like:

ProfileInfo {

   String getName();

   List<Map> getMaps();

   List<ServiceInfo> getServices();

}

I think it would also make sense if we want to start attaching other bits of "virtual configuration". I am thinking things here like logging, contact info, etc....

So profile complement maps by sitting on the service side of the fence?
Observations:
- not sure why we'd want multiple maps per profile. 1-1 relantionship
   would seem more natural to me
- actually a profile should be able to override most of the
   services, so it seems a good idea to have service overrides,
   looks better than having a list of srs in the MapInfo.
- what about always having a profile around, and name one the "default"
   just like we said we would have had a default map? This might reduce
   the number of special case handling in the code (e.g., no services
   lingering in the void without being attached to a profile, same
   goes for the "default" map).

Wondering how the overrides would play thought? Say you have a basic
WFS setup. And that for a profile you want to override it.
So, is the override working on an attribute by attribute basis?
Or you copy over the default wfs service info and from there on
the two live separate lives?
The latter seems to be easier to configure to me.

How would profiles be shown in the UI? They end up influencing
what you get when you click on a service configuration, but
also the layer list (stores and workspaces would be unaffected,
maybe styles too?).
It seems that this has to be done in some global, prominent setting,
a context of sort. Could be just a drop down at the top of
the left menu allowing you to choose the "active" profile (the one
being edited).

Another thing that's bugging me is that there is a list of things
that do pertain to configuration that are still left out,
think security, srs handling. They would need to be part of
the profile as well, and imho, participate as first level
elements in the config of GeoServer. So far they have been left
out because it was easier to code them up as properties files
when they were introduced, but they should come back so that we
have something we can call a "config", now what we have is kind
of a set of unrelated pieces.

Cheers
Andrea

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Hi all,

I have been thinking about virtual configurations and resource
publishing split at the data structure level. Up until we have MapInfo:

MapInfo {

   String getName();

   List<LayerInfo> getLayers();

   List<StyleInfo> getStyles();

   List<String> getSRSs();

   ...
}

Not sure about styles and srs... SRS are part of the WMS configuration,
and styles WMS wise are simply not something that stands up on its own.
I guess by that you're trying to limit the styles that one can see
by doing direct access to them in styles=...?

Hmmm... sure you it only applies to WMS. As of WFS 1.1 SRS's are into the mix for WFS as well as the capabilities document advertises which SRS's apply to a feature type.

I can see the argument for styles however. That said we have modeled styles as a first level construct in the catalog, so I was more or less continuing on that path. But yeah, including styles is not really a strong case... ok with leaving it out.

Which gives us basically the data side of the equation, but not the
services. One of the things undoubtedly people will want to do with a
virtual configuration is limit the services that can access a map. Now
we could do this:

MapInfo {

   List<ServiceInfo> getServices()

}

But this kind of seems off. Mainly because "map" lives on the catalog side of the fence, and services are a "configuration" entity. So the thought occurred to me to come up with another interface. For lack of a better name I am calling it a "profile". It would look something like:

ProfileInfo {

   String getName();

   List<Map> getMaps();

   List<ServiceInfo> getServices();

}

I think it would also make sense if we want to start attaching other bits of "virtual configuration". I am thinking things here like logging, contact info, etc....

So profile complement maps by sitting on the service side of the fence?
Observations:
- not sure why we'd want multiple maps per profile. 1-1 relantionship
  would seem more natural to me

Yeah I thought about this too. I guess my rationale was that multiple maps would be more flexible. To illustrate my train of thought I am considering a "production" profile and a "development" profile. The production one would want to be tweaked out, production logging, etc.. etc.. and maybe the development one not. Within those profiles you may wish to have multiple maps, maybe one for publishing vector data, and another for publishing WMS...

Anyways, it would be a shear convenience as the only real purpose it would serve would be to create some configurations that could be shared and to perhaps prevent some duplication.

I would be all for at this point keeping things simple and leaving the relationship as 1-1.

- actually a profile should be able to override most of the
  services, so it seems a good idea to have service overrides,
  looks better than having a list of srs in the MapInfo.
- what about always having a profile around, and name one the "default"
  just like we said we would have had a default map? This might reduce
  the number of special case handling in the code (e.g., no services
  lingering in the void without being attached to a profile, same
  goes for the "default" map).

Sounds like a good idea to me.

Wondering how the overrides would play thought? Say you have a basic
WFS setup. And that for a profile you want to override it.
So, is the override working on an attribute by attribute basis?
Or you copy over the default wfs service info and from there on
the two live separate lives?
The latter seems to be easier to configure to me.

Yes, the latter was what I had in mind as well.

How would profiles be shown in the UI? They end up influencing
what you get when you click on a service configuration, but
also the layer list (stores and workspaces would be unaffected,
maybe styles too?).
It seems that this has to be done in some global, prominent setting,
a context of sort. Could be just a drop down at the top of
the left menu allowing you to choose the "active" profile (the one
being edited).

Good point. Not sure what the best way about going about this is. The drop down for active profile seems nice as it does not require us to deal with nesting, ie a top level profiles and then drill down into services, etc...

I am however curious to hear what the designers think about this one.

Another thing that's bugging me is that there is a list of things
that do pertain to configuration that are still left out,
think security, srs handling. They would need to be part of
the profile as well, and imho, participate as first level
elements in the config of GeoServer. So far they have been left
out because it was easier to code them up as properties files
when they were introduced, but they should come back so that we
have something we can call a "config", now what we have is kind
of a set of unrelated pieces.

+1. We need to come up with interfaces in our model for these guys.

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Justin Deoliveira ha scritto:

Not sure about styles and srs... SRS are part of the WMS configuration,
and styles WMS wise are simply not something that stands up on its own.
I guess by that you're trying to limit the styles that one can see
by doing direct access to them in styles=...?

Hmmm... sure you it only applies to WMS. As of WFS 1.1 SRS's are into the mix for WFS as well as the capabilities document advertises which SRS's apply to a feature type.

Hmmm.. yep. Yet, we don't advertise that information, since we'd end
up listing 3000+ srs for each layer, making a huuuge caps document :slight_smile:
Maybe we could list the srs for each layer only if the user provided
a limited list thought, that would give a reason to have a global
SRS list.

Too bad wfs does not have a layer nesting concept, it would allow
for more uniformity with WMS....

I can see the argument for styles however. That said we have modeled styles as a first level construct in the catalog, so I was more or less continuing on that path. But yeah, including styles is not really a strong case... ok with leaving it out.

Mine was more of a question than a -1 on it. Do we want to limit
direct access to named styles, or not? Can it be seen as a matter
of stability, like, a style so expensive that we want to provide
access to it only throught a specific Map that has been secured?

I think it would also make sense if we want to start attaching other bits of "virtual configuration". I am thinking things here like logging, contact info, etc....

So profile complement maps by sitting on the service side of the fence?
Observations:
- not sure why we'd want multiple maps per profile. 1-1 relantionship
  would seem more natural to me

Yeah I thought about this too. I guess my rationale was that multiple maps would be more flexible. To illustrate my train of thought I am considering a "production" profile and a "development" profile. The production one would want to be tweaked out, production logging, etc.. etc.. and maybe the development one not. Within those profiles you may wish to have multiple maps, maybe one for publishing vector data, and another for publishing WMS...

Oh, right, I did not thing about this case... makes a lot of sense.
The way the profile class is setup made me think Profile owns the
Maps, as opposed to Maps sharing the profile.
But I guess it can't be helped, since Profile is not part of the
catalog, whilst Map would be?

Anyways, thanks for the clarification, I think multiple maps
per profile is a good idea.

How would profiles be shown in the UI? They end up influencing
what you get when you click on a service configuration, but
also the layer list (stores and workspaces would be unaffected,
maybe styles too?).
It seems that this has to be done in some global, prominent setting,
a context of sort. Could be just a drop down at the top of
the left menu allowing you to choose the "active" profile (the one
being edited).

Good point. Not sure what the best way about going about this is. The drop down for active profile seems nice as it does not require us to deal with nesting, ie a top level profiles and then drill down into services, etc...

And thus once established the profile you'd also only see the
maps that are inside of it, right? Can a map be moved from one profile
to the other? Shared? (the latter, maybe not, would make for a many
to many relationship and we'd have to come up with a name to identify
the profile-map couple).

Cheers
Andrea

Oh, right, I did not thing about this case... makes a lot of sense.
The way the profile class is setup made me think Profile owns the
Maps, as opposed to Maps sharing the profile.
But I guess it can't be helped, since Profile is not part of the
catalog, whilst Map would be?

Anyways, thanks for the clarification, I think multiple maps
per profile is a good idea.

Thinking more about what a profile actually is. It sort of seems rather than come up with a new class... a profile is really just another instance of GeoServer. I mean... GeoServer would the super set of everything that a profile may want to change right (realizing that it does not store any maps at the moment).

So maybe that is how we want to approach this? Just a thought.

How would profiles be shown in the UI? They end up influencing
what you get when you click on a service configuration, but
also the layer list (stores and workspaces would be unaffected,
maybe styles too?).
It seems that this has to be done in some global, prominent setting,
a context of sort. Could be just a drop down at the top of
the left menu allowing you to choose the "active" profile (the one
being edited).

Good point. Not sure what the best way about going about this is. The drop down for active profile seems nice as it does not require us to deal with nesting, ie a top level profiles and then drill down into services, etc...

And thus once established the profile you'd also only see the
maps that are inside of it, right? Can a map be moved from one profile
to the other? Shared? (the latter, maybe not, would make for a many
to many relationship and we'd have to come up with a name to identify
the profile-map couple).

Thinking a bit more about this. How do we deal with multiple access. For a service call it seems rather simple, the thread local approach would do it. But what about the UI? I guess we would have to store the active profile via a session? Not exactly sure how this would look but seems necessary to accommodate multiple users editing different profiles simultaneously.

Cheers
Andrea

------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Justin Deoliveira ha scritto:

Oh, right, I did not thing about this case... makes a lot of sense.
The way the profile class is setup made me think Profile owns the
Maps, as opposed to Maps sharing the profile.
But I guess it can't be helped, since Profile is not part of the
catalog, whilst Map would be?

Anyways, thanks for the clarification, I think multiple maps
per profile is a good idea.

Thinking more about what a profile actually is. It sort of seems rather than come up with a new class... a profile is really just another instance of GeoServer. I mean... GeoServer would the super set of everything that a profile may want to change right (realizing that it does not store any maps at the moment).

Indeed I believe in the end that's the essence of what we're trying
to do here, that is, setup virtual GeoServer all running inside
the same webapp as opposed of having to deploy GeoServer n times,
the latter being hard to manage because of the extra memory
consumption, the lack of sharing at the data resource level (think
jdbc/sde connection pools) not to mention the various caches we
have around.
So yeah, from where I stand a profile really is "another geoserver",
but lighter. Think threads vs processes.

So maybe that is how we want to approach this? Just a thought.

How would profiles be shown in the UI? They end up influencing
what you get when you click on a service configuration, but
also the layer list (stores and workspaces would be unaffected,
maybe styles too?).
It seems that this has to be done in some global, prominent setting,
a context of sort. Could be just a drop down at the top of
the left menu allowing you to choose the "active" profile (the one
being edited).

Good point. Not sure what the best way about going about this is. The drop down for active profile seems nice as it does not require us to deal with nesting, ie a top level profiles and then drill down into services, etc...

And thus once established the profile you'd also only see the
maps that are inside of it, right? Can a map be moved from one profile
to the other? Shared? (the latter, maybe not, would make for a many
to many relationship and we'd have to come up with a name to identify
the profile-map couple).

Thinking a bit more about this. How do we deal with multiple access. For a service call it seems rather simple, the thread local approach would do it. But what about the UI? I guess we would have to store the active profile via a session? Not exactly sure how this would look but seems necessary to accommodate multiple users editing different profiles simultaneously.

Wicket has this type safe session concept that is store inside the
squishy http sessions, so yeah, that's what we should do.
Security wise this opens up the question of rigths to manage a certain
profile and not others (and possibly associated workspaces).
I guess ISP like settings would very much like this :wink:

Cheers
Andrea

Thinking more about what a profile actually is. It sort of seems rather
than come up with a new class... a profile is really just another
instance of GeoServer. I mean... GeoServer would the super set of
everything that a profile may want to change right (realizing that it
does not store any maps at the moment).

Indeed I believe in the end that's the essence of what we're trying
to do here, that is, setup virtual GeoServer all running inside
the same webapp as opposed of having to deploy GeoServer n times,
the latter being hard to manage because of the extra memory
consumption, the lack of sharing at the data resource level (think
jdbc/sde connection pools) not to mention the various caches we
have around.
So yeah, from where I stand a profile really is "another geoserver",
but lighter. Think threads vs processes.

So maybe that is how we want to approach this? Just a thought.

I like the thought (a lot). As noted we do want to share some things
if we can (datastore instances; jdbc connection pools, etc..).
One thing I am not clear on though is if we want to share feature
locks between geoserver instances (it makes sense in that we are
trying to reserve the same datasets; but it also presents a conflict
in that the lock may be managed by a different geoserver instance then
the one you are communicating with).

One thing that is missing from the "multiple geoservers" is the story
of switching between development and production profiles - I wondered
if we could just offer a way to save out your configuration allowing
the user to save out a production and development (and whatever)
configuration as needed?

Jody
Jody

Jody Garnett wrote:

Thinking more about what a profile actually is. It sort of seems rather
than come up with a new class... a profile is really just another
instance of GeoServer. I mean... GeoServer would the super set of
everything that a profile may want to change right (realizing that it
does not store any maps at the moment).

Indeed I believe in the end that's the essence of what we're trying
to do here, that is, setup virtual GeoServer all running inside
the same webapp as opposed of having to deploy GeoServer n times,
the latter being hard to manage because of the extra memory
consumption, the lack of sharing at the data resource level (think
jdbc/sde connection pools) not to mention the various caches we
have around.
So yeah, from where I stand a profile really is "another geoserver",
but lighter. Think threads vs processes.

So maybe that is how we want to approach this? Just a thought.

I like the thought (a lot). As noted we do want to share some things
if we can (datastore instances; jdbc connection pools, etc..).
One thing I am not clear on though is if we want to share feature
locks between geoserver instances (it makes sense in that we are
trying to reserve the same datasets; but it also presents a conflict
in that the lock may be managed by a different geoserver instance then
the one you are communicating with).

Hmm... that is an interesting point, one I had never thought of. However since the point of locking is to prevent concurrent modification of physical data I think it makes sense to continue to keep locks at that level. That said, this is has always been an issue with GeoServer. Since in the only locking implementation we have locks are stored on the database level, if you add the same postgis database twice as separate datastores you get a different set of locks. Which could probably be considered a bug.

I am also a bit confused by the terminology "geoserver instance". To me an "instance" == "physical server". So we are not talking about multiple instances correct? Just multiple views of a single instance?

One thing that is missing from the "multiple geoservers" is the story
of switching between development and production profiles - I wondered
if we could just offer a way to save out your configuration allowing
the user to save out a production and development (and whatever)
configuration as needed?

The devlopment vs production profile stuff was really just an example... a profile can really represent anything you want. About saving out, not sure I follow you? What you describe sounds like different data directories to me... saving out one configuration, restarting and switching to another. Not sure I follow.

Jody

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

I am also a bit confused by the terminology "geoserver instance". To me an
"instance" == "physical server". So we are not talking about multiple
instances correct? Just multiple views of a single instance?

I am going to try and clarify (this is only my understanding of
previous emails so I could be off base)...I am going to just use the
world "geoserver" (and not geoserver instance).

right now we have:
http://xxxxxxx/geoserver/wfs
http://xxxxxxx/geoserver/wms
http://xxxxxxx/geoserver/wcs

This parth /geoserver" is what people would think of as a "geoserver";
we have had the request to break out multiple wfs entry points (to
account for people with different security privileges etc...). Here is
an example with a "public" geoserver and one for "field" workers
collecting data.

http://xxxxxxx/geoserver/public/wfs
http://xxxxxxx/geoserver/public/wms
http://xxxxxxx/geoserver/public/wcs
http://xxxxxxx/geoserver/field/wfs
http://xxxxxxx/geoserver/field/wms
http://xxxxxxx/geoserver/field/wcs

I suspect that this could be represented as only represents two
geoservers; with different SERVICE=XXX entries.
http://xxxxxxx/geoserver/public?SERVICE=WMS+REQUEST=GetCapabilities
http://xxxxxxx/geoserver/field?SERVICE=WMS+REQUEST=GetCapabilities

We would like to ensure these two geoservers share a common pool of
datastore instances; if Just has split out the "catalog" data
structure from the pool of datastores this may in fact be easy to
accomplish; if the same datastore is configured on both the same
instance will be used for example?

One thing that is missing from the "multiple geoservers" is the story
of switching between development and production profiles - I wondered
if we could just offer a way to save out your configuration allowing
the user to save out a production and development (and whatever)
configuration as needed?

The devlopment vs production profile stuff was really just an example... a
profile can really represent anything you want. About saving out, not sure I
follow you? What you describe sounds like different data directories to
me... saving out one configuration, restarting and switching to another. Not
sure I follow.

The example made sense to me (switching between development and
production databases is a common requirement that right now people
manage by copying their entire data directory; or knowing which files
to replace in their existing data directory). You know how confluence
lets you export out the wiki as a zip of xml files? And then they
provide the facility to let you restore from an exported archive? I
was thinking if we had something like that we could let people keep
different configurations on hand. This may however not be a good
idea...

Jody

Jody Garnett ha scritto:

The example made sense to me (switching between development and
production databases is a common requirement that right now people
manage by copying their entire data directory; or knowing which files
to replace in their existing data directory). You know how confluence
lets you export out the wiki as a zip of xml files? And then they
provide the facility to let you restore from an exported archive? I
was thinking if we had something like that we could let people keep
different configurations on hand. This may however not be a good
idea...

I've seen people just keep the GeoServer configuration in svn.
When they are happy with it in the dev server the commit, eventually
tag, and svn up on the production server. Something goes wrong? Just
revert to the previous tagged version.
Brilliant, but probably a bit too hard for the average (GUI spoiled)
admin.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.