[Geoserver-devel] FM merge - when and what (wrt Grid Coverage Merge)

During the Grid Coverage/Renderer improvements IRC today, we talked about merging the grid coverage branch onto trunk.

See:
http://docs.codehaus.org/display/GEOTOOLS/2006/05/01/Renderer+IRC
and the " Renderer improvements meeting part 2 (Grid Coverge Merge to Trunk)" message thread.

I know almost nothing about the FM branch, and the last I heard was that it was going to be another 6 months before its done. Apparently, thats incorrect, and its going to be finished up next month (june) because there's a lot of OWS-4 work thats depending on it.

Perhaps we can get a check-in from the folks who are working on FM, or who are current stalled (or will soon be stalled) waiting for it - at the very least get a list of the FM 'stakeholders'.

The main problem is that a Grid Coverage merge could quite possibily "step on the toes" of the FM folks (see the IRC discussion linked above). Trying to do the Grid Coverage merge and FM merge at the same time is almost certainly going to cause problems, so it seems like we should either:

a) merge current trunk (I believe the main changes are changes to expression - correct me if I'm wrong) with Grid Coverage Branch + Renderering Improvments - this would be a 2.3 release.
   2.4 would then be 2.3 (ie. coverages) + FM.

OR

b) wait for the FM merge to occur (current trunk + FM = 2.3) then merge the Grid Coverage stuff in later (2.4 = trunk + FM + Grid).

There's other options, but I thought I would just throw this out so that the FM folks could start talking. Whats the plan? (In terms of those working on it, and those "waiting" on it) Is doing a Grid merge going to cause problems? If so, what do we do?

I dont think there's a pressing need "right now, today" to merge the grid stuff in (I could be wrong, though), but I know a lot of people have been waiting for it for a while.

dave

ps. Jody's kindly offered (free-as-in-beer-and-as-in-speech) FM walk-throughs all this week on IRC.
pps. There is starting to be a fairly large divergence between 2.2.x and trunk. This is making it difficult to move 2.2.x patches to trunk - not all bug fixes are finding there way to trunk. This seems to indicate that we should kick out a 2.3 fairly soon.
ppps. Justin has said that the FM branch is basically a re-write of the geotools core. It certainly makes sense to call the end result "geotools 3" to recognize this.

oops - meant to send this to geotools-devel not geoserver-devel.

If you're interested in FM or Grid Coverage (ie. WMS-with-raster or WCS), switch over to geotools-devel.

dave

I've been trying to get a handle/response re FM game plan for a while too. Its like herding Schrodinger's cats in the dark whilst enjoying the afterglow of a bottle of port. (from very vague memories.. )

From a "test-driven development" perspective I think that we will need to invest in a decent set of test cases for each data source. We are starting to get a useful set of test cases from the "complex-features" work but they need to be formalised into some of testing framework - at the moment there is a disconnect between data store tests and geoserver tests, it would be nice to have the data store test data suite exposable right through the geoserver functions, and to reuse the bulk of the test case code between datastore test suites. A common abstract data store test suite that is extended with initialisation and cleanup routines or something.

This is relevant now, since coverages may have a very different set of issues, and merging then refactoring may be more expensive that getting the test suite sorted out. The tests suite will need to exercise the FM concepts as far as I can see, since there are a range of critical issues relating to namespaces and building the internal feature type metadata that are integral to datastore behaviour.

So, from this pragmatic perspective, is it going to be more efficient to do the FM merge first, or the coverage?

Secondly, the relationship between coverages and features is quite strong - there will be cases where data stored in a grid format (like NETCDF) will contain features (imagine position being just a dependent variable) - so extracting for example a flight path or ship cruise from a grid format, as a feature, is a natural thing to do. Likewise, consider the common case of clicking on a raster derived map and returning a feature containing all the values in the grid at that point. Timeseries features are often stored in array coverages.

The merging of coverage concepts _without_ a feature model may result in a lot of in-built limitations in the architecture that will be difficult and costly to fix. (The effort required to unpack the erroneous assumption that the underlying database schema could form the basis of an interoperable feature type has been enormous - and would have been much less with the blow-torch of analysis applied earlier.

I for one would like to see a common architecture for FM and coverages before merging them in. If this architecture identifies that they can be safely done independently, so well and good, but if not then I think we need to follow the underlying logic in our planning.

Specific questions I have relating to the project plan for FM:
a) What will be involved in migrating the complex-feature datastores to FM?
b) how will the project be managed? Will there be a single point of contact to find out what is going on?
c) what can I do to help?

Rob Atkinson

David Blasby wrote:

During the Grid Coverage/Renderer improvements IRC today, we talked about merging the grid coverage branch onto trunk.

See:
http://docs.codehaus.org/display/GEOTOOLS/2006/05/01/Renderer+IRC
and the " Renderer improvements meeting part 2 (Grid Coverge Merge to Trunk)" message thread.

I know almost nothing about the FM branch, and the last I heard was that it was going to be another 6 months before its done. Apparently, thats incorrect, and its going to be finished up next month (june) because there's a lot of OWS-4 work thats depending on it.

Perhaps we can get a check-in from the folks who are working on FM, or who are current stalled (or will soon be stalled) waiting for it - at the very least get a list of the FM 'stakeholders'.
The main problem is that a Grid Coverage merge could quite possibily "step on the toes" of the FM folks (see the IRC discussion linked above). Trying to do the Grid Coverage merge and FM merge at the same time is almost certainly going to cause problems, so it seems like we should either:

a) merge current trunk (I believe the main changes are changes to expression - correct me if I'm wrong) with Grid Coverage Branch + Renderering Improvments - this would be a 2.3 release.
  2.4 would then be 2.3 (ie. coverages) + FM.

OR

b) wait for the FM merge to occur (current trunk + FM = 2.3) then merge the Grid Coverage stuff in later (2.4 = trunk + FM + Grid).

There's other options, but I thought I would just throw this out so that the FM folks could start talking. Whats the plan? (In terms of those working on it, and those "waiting" on it) Is doing a Grid merge going to cause problems? If so, what do we do?

I dont think there's a pressing need "right now, today" to merge the grid stuff in (I could be wrong, though), but I know a lot of people have been waiting for it for a while.

dave

ps. Jody's kindly offered (free-as-in-beer-and-as-in-speech) FM walk-throughs all this week on IRC.
pps. There is starting to be a fairly large divergence between 2.2.x and trunk. This is making it difficult to move 2.2.x patches to trunk - not all bug fixes are finding there way to trunk. This seems to indicate that we should kick out a 2.3 fairly soon.
ppps. Justin has said that the FM branch is basically a re-write of the geotools core. It certainly makes sense to call the end result "geotools 3" to recognize this.

-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Hi guys,

I know communication about fm has been slim. But please keep in mind the only reason we can even consider moving on fm is because certain people have spent *volunteer* time to get it into such a shape, which in my view is a failure in the management process of the project. And please dont take this as pointing a finger, because i am to blame as much as anyone else. But in defense there has been lots of discussion about certain issues with fm the geotools and geoapi lists and the only people that have cared are Jody and Bryce. So I turn it around on other people to say they haven't been exactly interested in it which doesn't exactly facilitate communication. People have just been expecting it to work. And unless I am mistaken this community decided to go forward with this long before I even became an active member. Perhaps things have changed and this needs to be revisited and in PR's words perform a formal cost benefit analysys. But that is something for the PMC to decide on.

That being said, here are where things are at:

We have api, main, referencing, coverage building (however not passing all test cases). All the xml, data (as in DataStore), and jdbc stuff has been ripped out of main.

Now I think the timings "6 months" vs "1 month" need a bit of clarification. My original estimate of 6 months was to get geotools based on FM to the state where geotools is right now. IE relatively stable and well tested (with *all* data stores working).

And IMHO, 6 months is very optimistic unless we can actually get more people vested in seeing this thing go. With everyone worried about getting other applications going which just use geotools and willing to hack around certain problems.

So here is what I see possible in one month (and again with more manpower):

1. Feature model turning over (ie. all major bugs fixed and we are actually able to create usable features).

2. Filter / expression subsystem ported (most of this is already done)

3. Data module ported.

4. 1 or 2 data stores ported over (postgis and shapefile for instance).

Again I say this is optimistic because as of right now only 1/2 of a single developers time is devoted toward this. However I know that Gabriel and Jody are also willing to kick in some time toward this which makes me optimistic.

I apologize if I seem pessimistic about this but I really think people need to know where things are at with fm. It is a rewrite of the core of geotools so I don't see how much else can be expected. IMHO fm cant be involved in another 2.x release, it needs to be called Geotools 3, if for any reason to drive home the fact that it is drastically different, to users and to developers.

Ok, sorry for that, only good vibes. The list above I believe is enough for OWS-4, and most importantly, enough to do it right. FM is a huge step forward in terms of fixing some of the major problems with geotools. And with OWS-4 driving things, what we get it is going to be well tested with most of the major kinks done right.

-Justin

David Blasby wrote:

During the Grid Coverage/Renderer improvements IRC today, we talked about merging the grid coverage branch onto trunk.

See:
http://docs.codehaus.org/display/GEOTOOLS/2006/05/01/Renderer+IRC
and the " Renderer improvements meeting part 2 (Grid Coverge Merge to Trunk)" message thread.

I know almost nothing about the FM branch, and the last I heard was that it was going to be another 6 months before its done. Apparently, thats incorrect, and its going to be finished up next month (june) because there's a lot of OWS-4 work thats depending on it.

Perhaps we can get a check-in from the folks who are working on FM, or who are current stalled (or will soon be stalled) waiting for it - at the very least get a list of the FM 'stakeholders'.
The main problem is that a Grid Coverage merge could quite possibily "step on the toes" of the FM folks (see the IRC discussion linked above). Trying to do the Grid Coverage merge and FM merge at the same time is almost certainly going to cause problems, so it seems like we should either:

a) merge current trunk (I believe the main changes are changes to expression - correct me if I'm wrong) with Grid Coverage Branch + Renderering Improvments - this would be a 2.3 release.
  2.4 would then be 2.3 (ie. coverages) + FM.

OR

b) wait for the FM merge to occur (current trunk + FM = 2.3) then merge the Grid Coverage stuff in later (2.4 = trunk + FM + Grid).

There's other options, but I thought I would just throw this out so that the FM folks could start talking. Whats the plan? (In terms of those working on it, and those "waiting" on it) Is doing a Grid merge going to cause problems? If so, what do we do?

I dont think there's a pressing need "right now, today" to merge the grid stuff in (I could be wrong, though), but I know a lot of people have been waiting for it for a while.

dave

ps. Jody's kindly offered (free-as-in-beer-and-as-in-speech) FM walk-throughs all this week on IRC.
pps. There is starting to be a fairly large divergence between 2.2.x and trunk. This is making it difficult to move 2.2.x patches to trunk - not all bug fixes are finding there way to trunk. This seems to indicate that we should kick out a 2.3 fairly soon.
ppps. Justin has said that the FM branch is basically a re-write of the geotools core. It certainly makes sense to call the end result "geotools 3" to recognize this.

--
Justin Deoliveira
jdeolive@anonymised.com
The Open Planning Project
http://topp.openplans.org

We have api, main, referencing, coverage building (however not passing all test cases).

Is this because:

a) the test cases themselves are not up to date with the FM (this was the case with most of the test cases I fixed for complex-features)
b) the core engineering is incomplete and the people involved in the refactoring are going to be required to complete and document work
or c) its the ancilliary functions (eg rendering) that need to be updated and this can be delegated out to more developers?

If the latter, there will be a point at which developers working on something against the legacy branches should start to look at porting and testing against FM as a matter of course, to bring them up to speed with geotools 3 as well as get the work done? When do we reach that point?

All the xml, data (as in DataStore), and jdbc stuff has been ripped out of main.

This sounds like a good thing - it feels like a lot of apples and pigs in a barrel at the moment - but does ripped out mean refactored into other packages or just ignored for now? I'm all in favour of creating an abstract data store and a set of data store basic test cases that can be inherited by all data stores.

Ok, sorry for that, only good vibes. The list above I believe is enough for OWS-4, and most importantly, enough to do it right. FM is a huge step forward in terms of fixing some of the major problems with geotools. And with OWS-4 driving things, what we get it is going to be well tested with most of the major kinks done right.

So for OWS4 there will need to be a costed plan to get it working. This week!

-Justin

David Blasby wrote:

During the Grid Coverage/Renderer improvements IRC today, we talked about merging the grid coverage branch onto trunk.

See:
http://docs.codehaus.org/display/GEOTOOLS/2006/05/01/Renderer+IRC
and the " Renderer improvements meeting part 2 (Grid Coverge Merge to Trunk)" message thread.

I know almost nothing about the FM branch, and the last I heard was that it was going to be another 6 months before its done. Apparently, thats incorrect, and its going to be finished up next month (june) because there's a lot of OWS-4 work thats depending on it.

Perhaps we can get a check-in from the folks who are working on FM, or who are current stalled (or will soon be stalled) waiting for it - at the very least get a list of the FM 'stakeholders'.
The main problem is that a Grid Coverage merge could quite possibily "step on the toes" of the FM folks (see the IRC discussion linked above). Trying to do the Grid Coverage merge and FM merge at the same time is almost certainly going to cause problems, so it seems like we should either:

a) merge current trunk (I believe the main changes are changes to expression - correct me if I'm wrong) with Grid Coverage Branch + Renderering Improvments - this would be a 2.3 release.
  2.4 would then be 2.3 (ie. coverages) + FM.

OR

b) wait for the FM merge to occur (current trunk + FM = 2.3) then merge the Grid Coverage stuff in later (2.4 = trunk + FM + Grid).

There's other options, but I thought I would just throw this out so that the FM folks could start talking. Whats the plan? (In terms of those working on it, and those "waiting" on it) Is doing a Grid merge going to cause problems? If so, what do we do?

I dont think there's a pressing need "right now, today" to merge the grid stuff in (I could be wrong, though), but I know a lot of people have been waiting for it for a while.

dave

ps. Jody's kindly offered (free-as-in-beer-and-as-in-speech) FM walk-throughs all this week on IRC.
pps. There is starting to be a fairly large divergence between 2.2.x and trunk. This is making it difficult to move 2.2.x patches to trunk - not all bug fixes are finding there way to trunk. This seems to indicate that we should kick out a 2.3 fairly soon.
ppps. Justin has said that the FM branch is basically a re-write of the geotools core. It certainly makes sense to call the end result "geotools 3" to recognize this.