[Geoserver-devel] GSIP 31 - Use DataAccess API - Revised and Implemented

I have revised GSIP 31 to include substantial details of the changes proposed. I call on PSC members to vote on the proposal.
http://geoserver.org/display/GEOS/GSIP+31+-+Use+DataAccess+API

The implementation is complete. There is one patch for each of the five affected modules, each attached to its own Jira issue; these are subtasks of GEOS-2568:
http://jira.codehaus.org/browse/GEOS-2568

I have not assigned these issues. Will someone from TOPP please assign these subtasks for patch review?

All requisite GeoTools changes have been committed to trunk (thanks Jody).

A WFS GetFeature unit test (SampleDataAccessGetFeatureTest in community/app-schema/sample-data-access-test) demonstrates the successful operation of the GSIP 31 implementation. The test does not yet pass because there is no encoder configuration capable of handling complex features, but examination of its execution shows that the correct features are retrieved and presented for encoding. A test output format will be used to get this test going very soon. A complex feature encoder configuration will be the next work. These are outside the scope of GSIP 31.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

Thanks Ben this appears to be all in order.

I have revised GSIP 31 to include substantial details of the changes proposed. I call on PSC members to vote on the proposal.
http://geoserver.org/display/GEOS/GSIP+31±+Use+DataAccess+API

+1

The implementation is complete. There is one patch for each of the five affected modules, each attached to its own Jira issue; these are subtasks of GEOS-2568:
http://jira.codehaus.org/browse/GEOS-2568

This is exciting; I see you have patches for the wfs and wms module already (last I checked you were just doing wfs module as a test of completeness).

Or perhaps these are only patches to make it compile? I cannot tell from the patch description.

I have not assigned these issues. Will someone from TOPP please assign these subtasks for patch review?

All requisite GeoTools changes have been committed to trunk (thanks Jody).

A WFS GetFeature unit test (SampleDataAccessGetFeatureTest in community/app-schema/sample-data-access-test) demonstrates the successful operation of the GSIP 31 implementation. The test does not yet pass because there is no encoder configuration capable of handling complex features, but examination of its execution shows that the correct features are retrieved and presented for encoding. A test output format will be used to get this test going very soon. A complex feature encoder configuration will be the next work. These are outside the scope of GSIP 31.

That makes it clear; thanks for the clarification.

When it comes time to do the rich feature encoder you have a couple of options (ranging from a XMLTransformer similar to the current GML2 approach; or using the xml binding framework - have you made any decisions about direction?

Jody

Jody Garnett wrote:

This is exciting; I see you have patches for the wfs and wms module already (last I checked you were just doing wfs module as a test of completeness).
Or perhaps these are only patches to make it compile? I cannot tell from the patch description.

The main reason for the patches is to get everything to compile and all the existing unit tests to pass. I then made further changes to get WFS to work up the the encoder problem. I have not tested WMS with DataAccess, nor am I likely to. The first step (GSIP 31) is to just open the core catalog API and implementation to support DataAccess.

When it comes time to do the rich feature encoder you have a couple of options (ranging from a XMLTransformer similar to the current GML2 approach; or using the xml binding framework - have you made any decisions about direction?

I was going to see how far I got extending the existing configuration with additional binding overrides. This was the approach taken in community-schemas in 1.6.x with great success, and has the advantage of maximising reuse.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

+1 here

-------------------------------------------------------
Eng. Alessio Fabiani
Vice-President /CTO GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584 980933
fax: +39 0584 983027
mob: +39 349 8227000

http://www.geo-solutions.it

-------------------------------------------------------

On Thu, Jan 29, 2009 at 4:53 AM, Ben Caradoc-Davies
<Ben.Caradoc-Davies@anonymised.com> wrote:

Jody Garnett wrote:

This is exciting; I see you have patches for the wfs and wms module already (last I checked you were just doing wfs module as a test of completeness).
Or perhaps these are only patches to make it compile? I cannot tell from the patch description.

The main reason for the patches is to get everything to compile and all
the existing unit tests to pass. I then made further changes to get WFS
to work up the the encoder problem. I have not tested WMS with
DataAccess, nor am I likely to. The first step (GSIP 31) is to just open
the core catalog API and implementation to support DataAccess.

When it comes time to do the rich feature encoder you have a couple of options (ranging from a XMLTransformer similar to the current GML2 approach; or using the xml binding framework - have you made any decisions about direction?

I was going to see how far I got extending the existing configuration
with additional binding overrides. This was the approach taken in
community-schemas in 1.6.x with great success, and has the advantage of
maximising reuse.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Ben Caradoc-Davies wrote:

Jody Garnett wrote:

This is exciting; I see you have patches for the wfs and wms module already (last I checked you were just doing wfs module as a test of completeness).
Or perhaps these are only patches to make it compile? I cannot tell from the patch description.

The main reason for the patches is to get everything to compile and all the existing unit tests to pass. I then made further changes to get WFS to work up the the encoder problem. I have not tested WMS with DataAccess, nor am I likely to. The first step (GSIP 31) is to just open the core catalog API and implementation to support DataAccess.

Good work Ben. This is more along the lines of what i was expecting, a "spike", a single patch for just wfs would not be that.

One thing you have stated worries me, no testing of WMS!!?? Who are you expecting to test that? What is the regression testing strategy?

For the wfs encoding, what is your plan to move forward? As i see it the way things sit now wfs does not really function?

Anyways, i hope to try to apply the those patches to a fresh trunk checkout today, and try to give you more specific feedback.

When it comes time to do the rich feature encoder you have a couple of options (ranging from a XMLTransformer similar to the current GML2 approach; or using the xml binding framework - have you made any decisions about direction?

I was going to see how far I got extending the existing configuration with additional binding overrides. This was the approach taken in community-schemas in 1.6.x with great success, and has the advantage of maximising reuse.

Kind regards,

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Justin Deoliveira wrote:

One thing you have stated worries me, no testing of WMS!!?? Who are you expecting to test that? What is the regression testing strategy?

Release early and release often. :slight_smile:

Do the CITE tests cover WMS? Do the unit tests? I do not use WMS or WCS, so am unqualified to test them. I hope that Chris' proposed new release branch will stimulate a full regression test.

For the wfs encoding, what is your plan to move forward? As i see it the way things sit now wfs does not really function?

There is now a working unit test in community/app-schema/sample-data-access-test called SampleDataAccessGetFeatureTextListTest. This is skipped in maven because it will not run until the GSIP 31 patches are committed. With the patches, it will pass, producing a text response to a WFS request. The text-feature-list WFS output format (also hiding out in community/app-schema) returns a list of the type and fid of all features, and a count. This test confirms that all but the encoding is working.

My next tasks are to:

(1) Implement a WFSConfiguration that can encode complex features. There are many details to be handled such as workarounds for the GeoAPI complexType with simpleContent problem. I hope to largely reuse exisiting code.

(2) Find the cleanest way of integrating the the new configuration into GeoServer. This may require a patch to wfs GML3OutputFormat.

There is a full GetFeature WFS test (expecting XML) hiding in community/app-schema/sample-data-access-test, waiting for the encoder to be written.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

On Thursday 29 January 2009 01:53:09 Ben Caradoc-Davies wrote:

Jody Garnett wrote:

This is exciting; I see you have patches for the wfs and wms module

already (last I checked you were just doing wfs module as a test of

completeness). Or perhaps these are only patches to make it compile? I

cannot tell from the patch description.

The main reason for the patches is to get everything to compile and all

the existing unit tests to pass. I then made further changes to get WFS

to work up the the encoder problem. I have not tested WMS with

DataAccess, nor am I likely to. The first step (GSIP 31) is to just open

the core catalog API and implementation to support DataAccess.

When it comes time to do the rich feature encoder you have a couple of

options (ranging from a XMLTransformer similar to the current GML2

approach; or using the xml binding framework - have you made any

decisions about direction?

I was going to see how far I got extending the existing configuration

with additional binding overrides. This was the approach taken in

community-schemas in 1.6.x with great success, and has the advantage of

maximising reuse.

hopefully with the integration effort they shouldn’t need to be binding overrides at all? at least a handy of them should become the de-facto bindings I guess, or am I missing something?

cheers,

Gabriel

Kind regards,

Gabriel Roldan

OpenGeo - http://www.opengeo.org

Ben Caradoc-Davies wrote:

Justin Deoliveira wrote:

One thing you have stated worries me, no testing of WMS!!?? Who are you expecting to test that? What is the regression testing strategy?

Release early and release often. :slight_smile:

Do the CITE tests cover WMS? Do the unit tests? I do not use WMS or WCS, so am unqualified to test them. I hope that Chris' proposed new release branch will stimulate a full regression test.

I misunderstood the initial statement but Rob cleared things up. I thought it meant that no backwards compatibility was tested. But after looking at the patch and applying it is clear that wms continues to function with datastore. That is good enough for me.

For the wfs encoding, what is your plan to move forward? As i see it the way things sit now wfs does not really function?

There is now a working unit test in community/app-schema/sample-data-access-test called SampleDataAccessGetFeatureTextListTest. This is skipped in maven because it will not run until the GSIP 31 patches are committed. With the patches, it will pass, producing a text response to a WFS request. The text-feature-list WFS output format (also hiding out in community/app-schema) returns a list of the type and fid of all features, and a count. This test confirms that all but the encoding is working.

My next tasks are to:

(1) Implement a WFSConfiguration that can encode complex features. There are many details to be handled such as workarounds for the GeoAPI complexType with simpleContent problem. I hope to largely reuse exisiting code.

(2) Find the cleanest way of integrating the the new configuration into GeoServer. This may require a patch to wfs GML3OutputFormat.

There is a full GetFeature WFS test (expecting XML) hiding in community/app-schema/sample-data-access-test, waiting for the encoder to be written.

Sounds like a good path. Let me know if you have any problems or questinos with the encoder.

So in terms of the proposal I guess there will be multiple phases... with voting at each round. This being phase 1.

Kind regards,

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

Gabriel Roldan wrote:

On Thursday 29 January 2009 01:53:09 Ben Caradoc-Davies wrote:

I was going to see how far I got extending the existing configuration
with additional binding overrides. This was the approach taken in
community-schemas in 1.6.x with great success, and has the advantage of
maximising reuse.

hopefully with the integration effort they shouldn't need to be binding overrides at all? at least a handy of them should become the de-facto bindings I guess, or am I missing something?

The ideal case would be to not need any overrides. At the moment, WFSConfiguration does not work with complex features. If this could be fixed with changes to the existing bindings, then no additional encoder configuration would be required. However, because GeoAPI cannot represent complexType-with-simpleContent (app-schema smuggles the simple content in a simpleContent property), at least one binding override will be required, hence a new configuration.

My aim is to fix existing code as much as possible, and make the new configuration as small as possible; in the future it need not exist.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

I think we should feel free to fix issues in GeoAPI, - i.e. simplify our problem by submitting a change request. GeoAPI changes are expected by the maintainers AFAIK

Rob

-----Original Message-----
From: Caradoc-Davies, Ben (E&M, Kensington)
Sent: Monday, 2 February 2009 11:18 AM
To: Gabriel Roldan
Cc: Jody Garnett; Geoserver-devel; Gabriel Roldan; Atkinson, Rob (CLW, Lucas Heights); Justin Deoliveira; Andrea Aime
Subject: Re: GSIP 31 - Use DataAccess API - Revised and Implemented

Gabriel Roldan wrote:

On Thursday 29 January 2009 01:53:09 Ben Caradoc-Davies wrote:

I was going to see how far I got extending the existing configuration
with additional binding overrides. This was the approach taken in
community-schemas in 1.6.x with great success, and has the advantage of
maximising reuse.

hopefully with the integration effort they shouldn't need to be binding overrides at all? at least a handy of them should become the de-facto bindings I guess, or am I missing something?

The ideal case would be to not need any overrides. At the moment,
WFSConfiguration does not work with complex features. If this could be
fixed with changes to the existing bindings, then no additional encoder
configuration would be required. However, because GeoAPI cannot
represent complexType-with-simpleContent (app-schema smuggles the simple
content in a simpleContent property), at least one binding override will
be required, hence a new configuration.

My aim is to fix existing code as much as possible, and make the new
configuration as small as possible; in the future it need not exist.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

Indeed; this is a two way conversation.

However with this particular issue I feel we are missing the boat a little bit. You have a data structure that can represent an attribute with several pieces of formal information already - use it.

You can construct an ComplexAttribute when parsing and reverse the process when encoding. A ComplexAttribute holds two required pieces of information that you are going to have to give the model a name for (you would need to do this if you were drawing a UML diagram of your data structure anyways right?).

We can move this discussion to the GeoAPI list if you like; but so far I have not been convinced.

Jody
PS. You also have a clientProperties map that can be used to hold "informal" information if needed (such has hints to make your parser/encoder happy).
PPS. The one valid argument I thought through is XPath support; but I have yet to see it presented clearly

Rob.Atkinson@anonymised.com wrote:

I think we should feel free to fix issues in GeoAPI, - i.e. simplify our problem by submitting a change request. GeoAPI changes are expected by the maintainers AFAIK

Rob

  configuration would be required. However, because GeoAPI cannot represent complexType-with-simpleContent (app-schema smuggles the simple content in a simpleContent property), at least one binding override will be required, hence a new configuration.

My aim is to fix existing code as much as possible, and make the new configuration as small as possible; in the future it need not exist.

Kind regards,