[Geoserver-devel] Profiling Community Schema code in goeserver

GeoServer GeoServer Developer
1

As part of our robustness testing of AWDIP deployments of Geoserver (which includes Community Schemas), we have signed up to do "White box testing". Ie, work out what is happening inside Geoserver for the AWDIP use cases.
The end goal is to identify areas in the code where performance can be improved.

To this end, Stefan has started setting up profiling for Community Schema deploys of Geoserver. Unfortunately we have a limited budget for this and so we are looking for the 80% wins from 20% effort.

So what I'm interested to hear are suggestions as to what parts of the code we should focus on.

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing

--
Cameron Shorter
Geospatial Systems Architect
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Think Globally, Fix Locally
Commercial Support for Geospatial Open Source Solutions
http://www.lisasoft.com/LISAsoft/SupportedProducts.html

2

Cameron Shorter wrote:

As part of our robustness testing of AWDIP deployments of Geoserver (which includes Community Schemas), we have signed up to do "White box testing". Ie, work out what is happening inside Geoserver for the AWDIP use cases.
The end goal is to identify areas in the code where performance can be improved.

To this end, Stefan has started setting up profiling for Community Schema deploys of Geoserver. Unfortunately we have a limited budget for this and so we are looking for the 80% wins from 20% effort.

So what I'm interested to hear are suggestions as to what parts of the code we should focus on.
  

- Generating the capabilities document
- WMS Rendering speed is a good test; it manages to abuse just about everything.
- Straight up requesting Features based on BBox is good

Is this the kind of thing you had in mind? It matches the kind of activites uDig depends on :slight_smile:

Remember if you are actually testing community schema to make use of:
- When rendering WMS compare simple attribtues, vs styles that refer to nested attributes
- wfs requests that included all attributes, vs only some attribtues

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing
  

Have fun,
Jody

3

Good stuff.

Gabriel may have further insights, but here’s my perspective on what might be worth considering:

  1. No serious profiling work has been done on this build, and it wouldnt suprise me if significant gains can be made
  2. Fixes should target the trunk, so we’ll need to re-run the profiling against the trunk when we test the modules against the new plumbing
  3. The database seems to be hit twice per query, which may be a significant overhead if the db side is slow
  4. The Converter factory seems to be used inefficiently - although the architecture is flexible, in practice the required converter per attribute will be the same for each retrieved databse row, but we dont seem to be caching the choice and end up hitting all the factories in a loop each value.
  5. There is a “mutable attribute” for handling multi-valued properties without instantiating objects for every vale, but it still caches the result set from the JDBC database, leading to a scalability limitation. It would be better if this was streamed, but this is a significant rewrite worth considering only after getting stuff to trunk. It can handle quite a few records as is, but does not max out gracefully yet.

Rob

On Fri, May 16, 2008 at 10:10 AM, Cameron Shorter <cameron.shorter@anonymised.com> wrote:

As part of our robustness testing of AWDIP deployments of Geoserver
(which includes Community Schemas), we have signed up to do “White box
testing”. Ie, work out what is happening inside Geoserver for the AWDIP
use cases.
The end goal is to identify areas in the code where performance can be
improved.

To this end, Stefan has started setting up profiling for Community
Schema deploys of Geoserver. Unfortunately we have a limited budget for
this and so we are looking for the 80% wins from 20% effort.

So what I’m interested to hear are suggestions as to what parts of the
code we should focus on.

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing

–
Cameron Shorter
Geospatial Systems Architect
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Think Globally, Fix Locally
Commercial Support for Geospatial Open Source Solutions
http://www.lisasoft.com/LISAsoft/SupportedProducts.html

This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

4

Oops, last email was CCed to Openlayers, should have been CCed to dev@anonymised.com (a performance testing project being used for WFS testing)

Cameron Shorter wrote:

As part of our robustness testing of AWDIP deployments of Geoserver
(which includes Community Schemas), we have signed up to do "White box
testing". Ie, work out what is happening inside Geoserver for the AWDIP
use cases.
The end goal is to identify areas in the code where performance can be
improved.

To this end, Stefan has started setting up profiling for Community
Schema deploys of Geoserver. Unfortunately we have a limited budget for
this and so we are looking for the 80% wins from 20% effort.

So what I'm interested to hear are suggestions as to what parts of the
code we should focus on.

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing

--
Cameron Shorter
Geospatial Systems Architect
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Think Globally, Fix Locally
Commercial Support for Geospatial Open Source Solutions
http://www.lisasoft.com/LISAsoft/SupportedProducts.html

--
Cameron Shorter
Geospatial Systems Architect
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Think Globally, Fix Locally
Commercial Support for Geospatial Open Source Solutions
http://www.lisasoft.com/LISAsoft/SupportedProducts.html

5

WMS is not currently supported in this build., We decided to wait until the trunk was ready and port to it rather than clone and hack the WMS module.

(AWDIP services are about data transfer, not visualisation), they need to be judged against the requirements for this, so in fact blinding speed is not critical, but faithfulness to an advertised data standard and query capability is.

Rob

On Fri, May 16, 2008 at 10:43 AM, Jody Garnett <jgarnett@anonymised.com> wrote:

Cameron Shorter wrote:

As part of our robustness testing of AWDIP deployments of Geoserver
(which includes Community Schemas), we have signed up to do “White box
testing”. Ie, work out what is happening inside Geoserver for the AWDIP
use cases.
The end goal is to identify areas in the code where performance can be
improved.

To this end, Stefan has started setting up profiling for Community
Schema deploys of Geoserver. Unfortunately we have a limited budget for
this and so we are looking for the 80% wins from 20% effort.

So what I’m interested to hear are suggestions as to what parts of the
code we should focus on.

  • Generating the capabilities document
  • WMS Rendering speed is a good test; it manages to abuse just about
    everything.
  • Straight up requesting Features based on BBox is good

Is this the kind of thing you had in mind? It matches the kind of
activites uDig depends on :slight_smile:

Remember if you are actually testing community schema to make use of:

  • When rendering WMS compare simple attribtues, vs styles that refer to
    nested attributes
  • wfs requests that included all attributes, vs only some attribtues

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing

Have fun,
Jody

This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

6

Hi Cameron,

On 1.6.x there is the "stress" community module. It basically contains some code which uses a freemarker template to whip up a jmeter script which executes a bunch of random wms requests. We used it for our WMS performance benchmarking we did last year. Once you have the jmeter script set up all you do is run geoserver in a profiler (i suggest yourkit) and execute the jmeter scripts.

The set upi could be adapted without too much to WFS. Here is the link:

http://svn.codehaus.org/geoserver/branches/1.6.x/geoserver/stress/

-Justin

Cameron Shorter wrote:

As part of our robustness testing of AWDIP deployments of Geoserver (which includes Community Schemas), we have signed up to do "White box testing". Ie, work out what is happening inside Geoserver for the AWDIP use cases.
The end goal is to identify areas in the code where performance can be improved.

To this end, Stefan has started setting up profiling for Community Schema deploys of Geoserver. Unfortunately we have a limited budget for this and so we are looking for the 80% wins from 20% effort.

So what I'm interested to hear are suggestions as to what parts of the code we should focus on.

We will be collating the results of our tests here:
https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing

--
Justin Deoliveira
The Open Planning Project
jdeolive@anonymised.com

7

Rob Atkinson wrote:

3) The database seems to be hit twice per query, which may be a significant overhead if the db side is slow

That was one thing I discovered when profiling the AWDIP Geoserver yesterday. I talked with Andrea about that and he explained to me that this is necessary to get the number of features and is also done in the "normal" Geoserver. However, there some ways to improve the performance in this point (e.g. done in the postgis datastore). Have such improvements been implemented for the community module?

I also discovered that the function org.geotools.feature.Name.getURI() is called very, very often in an AWDIP Geoserver (not that often in a normal Geoserver). Each time it is called a StringBuffer-object is created, one string appended and then the content of the buffer is returned by the function as String. Since this function is used that frequently, it might be worth to do some tuning here. Maybe it's possible to store the return value in a variable to use it later, if the function is called on the same objects. Or maybe at least the StringBuffer object can be reused.

cheers,
stefan

--
Stefan Hansen
Software Engineer
LISAsoft

---------------------------------------------------
Ph: +61 3 8680 3200 Fax: +61 3 8680 3299 Mob: +61
Level 9, 601 Bourke St, Melbourne Vic 3000
---------------------------------------------------

LISAsoft is part of the A2end Group of Companies
http://www.ardec.com.au
http://www.lisasoft.com
http://www.terrapages.com

8

thanks Stefan, look forward to a report, which we’ll put in the planning pipeline! I;'m rather hoping for some dumb stuff to emerge, that we can fix easily.

current performance seems adequate for data transfer use, WMS tends to be used more interactively, so it will be useful to apply the tools each development cycle.

Rob A

On Fri, May 16, 2008 at 4:08 PM, Stefan Hansen <stefan.hansen@anonymised.com> wrote:

Rob Atkinson wrote:

  1. The database seems to be hit twice per query, which may be a significant overhead if the db side is slow

That was one thing I discovered when profiling the AWDIP Geoserver yesterday. I talked with Andrea about that and he explained to me that this is necessary to get the number of features and is also done in the “normal” Geoserver. However, there some ways to improve the performance in this point (e.g. done in the postgis datastore). Have such improvements been implemented for the community module?

I also discovered that the function org.geotools.feature.Name.getURI() is called very, very often in an AWDIP Geoserver (not that often in a normal Geoserver). Each time it is called a StringBuffer-object is created, one string appended and then the content of the buffer is returned by the function as String. Since this function is used that frequently, it might be worth to do some tuning here. Maybe it’s possible to store the return value in a variable to use it later, if the function is called on the same objects. Or maybe at least the StringBuffer object can be reused.

cheers,
stefan

–
Stefan Hansen
Software Engineer
LISAsoft

Ph: +61 3 8680 3200 Fax: +61 3 8680 3299 Mob: +61
Level 9, 601 Bourke St, Melbourne Vic 3000

LISAsoft is part of the A2end Group of Companies
http://www.ardec.com.au
http://www.lisasoft.com
http://www.terrapages.com

9

Is all the community schema work still taking place on a forked module on 1.6.x? It seems like a bunch of effort is being put into it there, which will then all have to be ported? What readiness on trunk is being waited for? It has the new feature model on it, and we've done an alpha release. Is there a plan to get on trunk? Or maybe I missed it and it's already there?

Rob Atkinson wrote:

WMS is not currently supported in this build., We decided to wait until the trunk was ready and port to it rather than clone and hack the WMS module.

(AWDIP services are about data transfer, not visualisation), they need to be judged against the requirements for this, so in fact blinding speed is not critical, but faithfulness to an advertised data standard and query capability is.

Rob

On Fri, May 16, 2008 at 10:43 AM, Jody Garnett <jgarnett@anonymised.com <mailto:jgarnett@anonymised.com>> wrote:

    Cameron Shorter wrote:
     > As part of our robustness testing of AWDIP deployments of Geoserver
     > (which includes Community Schemas), we have signed up to do
    "White box
     > testing". Ie, work out what is happening inside Geoserver for the
    AWDIP
     > use cases.
     > The end goal is to identify areas in the code where performance
    can be
     > improved.
     >
     > To this end, Stefan has started setting up profiling for Community
     > Schema deploys of Geoserver. Unfortunately we have a limited
    budget for
     > this and so we are looking for the 80% wins from 20% effort.
     >
     > So what I'm interested to hear are suggestions as to what parts
    of the
     > code we should focus on.
     >
    - Generating the capabilities document
    - WMS Rendering speed is a good test; it manages to abuse just about
    everything.
    - Straight up requesting Features based on BBox is good

    Is this the kind of thing you had in mind? It matches the kind of
    activites uDig depends on :slight_smile:

    Remember if you are actually testing community schema to make use of:
    - When rendering WMS compare simple attribtues, vs styles that refer to
    nested attributes
    - wfs requests that included all attributes, vs only some attribtues
     > We will be collating the results of our tests here:
     >
    https://extranet.lisasoft.com/wiki/index.php/AWDIP_Software_Test_Results#Observations_from_White_Box_Testing
     >
    Have fun,
    Jody

    -------------------------------------------------------------------------
    This SF.net email is sponsored by: Microsoft
    Defy all challenges. Microsoft(R) Visual Studio 2008.
    http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
    _______________________________________________
    Geoserver-devel mailing list
    Geoserver-devel@lists.sourceforge.net
    <mailto:Geoserver-devel@lists.sourceforge.net>
    https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4005,482cde32217621137850744!

------------------------------------------------------------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

!DSPAM:4005,482cde32217621137850744!

------------------------------------------------------------------------

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4005,482cde32217621137850744!

10

Chris Holmes wrote:

Is all the community schema work still taking place on a forked module on 1.6.x? It seems like a bunch of effort is being put into it there, which will then all have to be ported? What readiness on trunk is being waited for? It has the new feature model on it, and we've done an alpha release. Is there a plan to get on trunk? Or maybe I missed it and it's already there?

I just told Rob that trunk was ready a few days ago Chris; he was not aware that Gabriel's proposal for a DataStore super class had gone through ... that was the last thing that was needed... so now we wait for an H2DataStore that can map external keys as associations - yum!
Jody

11

Chris Holmes wrote:

Is all the community schema work still taking place on a forked module on 1.6.x?

Yes.

It seems like a bunch of effort is being put into it there, which will then all have to be ported?

That is correct. And it sucks. We have hard deadlines that require delivery now and cannot wait for a port to trunk. Yes, this is making more work for ourselves. Yes, I will resist all attempts to add new features until the port to trunk is done. In some cases, my resistance is futile.

What readiness on trunk is being waited for? It has the new feature model on it, and we've done an alpha release.

I think the trunk is satisfactory. Trunk is not the problem.

Is there a plan to get on trunk? Or maybe I missed it and it's already there?

There is a stated goal. I am not sure if it has been published. There is no plan, per se. I will make enquiries and report back.

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

12

The plan I plan to plan (with the paymaster’s consent) is:

  1. Achieve basic WFS 1.1, GML 3.1 capabilities required, implementing test cases.

  2. Deploy to time-sensitive testbed activity

  3. Port to trunk, and debug against test cases and WMS capability

  4. Hopefully update deployments to include WMS functionality.

If #3 looks problematic, we may be required to fork WMS module as well.

and, we are all desperate to start working against trunk so we can do those key bits: performance tune and improve configuration process.

Rob A

On Wed, May 21, 2008 at 12:03 PM, Ben Caradoc-Davies Ben.Caradoc-Davies@anonymised.com wrote:

Chris Holmes wrote:

Is all the community schema work still taking place on a forked module on 1.6.x?

Yes.

It seems like a bunch of effort is being put into it there, which will then all have to be ported?

That is correct. And it sucks. We have hard deadlines that require delivery now and cannot wait for a port to trunk. Yes, this is making more work for ourselves. Yes, I will resist all attempts to add new features until the port to trunk is done. In some cases, my resistance is futile.

What readiness on trunk is being waited for? It has the new feature model on it, and we’ve done an alpha release.

I think the trunk is satisfactory. Trunk is not the problem.

Is there a plan to get on trunk? Or maybe I missed it and it’s already there?

There is a stated goal. I am not sure if it has been published. There is no plan, per se. I will make enquiries and report back.

–
Ben Caradoc-Davies Ben.Caradoc-Davies@anonymised.com
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

13

Ben Caradoc-Davies wrote:

Chris Holmes wrote:

Is all the community schema work still taking place on a forked module on 1.6.x?

[snip]

Is there a plan to get on trunk? Or maybe I missed it and it's already there?

There is a stated goal. I am not sure if it has been published. There is no plan, per se. I will make enquiries and report back.

It is a stated goal of the Spatial Information Services Stack (SISS) project that GeoServer community-schemas be ported to trunk. SISS is an NCRIS project http://www.ncris.dest.gov.au/ that will commence in July. It is anticipated that the port be completed this year. It is a goal of the SISS project that community-schemas become a fully-supported component of the GeoServer trunk. This cannot be achieved without the cooperation of the community.

Dr Robert Woodcock, Director of AuScope Grid http://www.auscope.org/ and Project Manager of SISS, approached OpenPlans regarding this project, and was offered a subscription to GeoServer Enterprise. This was not what he wanted. The SISS project can contribute effort to the port of community-schemas to trunk as a contributor to an Open Source project, not as a commercial customer. SISS is all about improving the capability of Open Source software to deliver spatial services. This software can then be adopted by others. SISS is an enabling project, not one that intends to deliver any data itself.

As of this morning, Robert is a subscriber to this list, and will no doubt be keenly following this discussion.

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineer, CSIRO Exploration and Mining
Australian Resources Research Centre
26 Dick Perry Ave, Kensington WA 6151, Australia

14

Ben Caradoc-Davies wrote:

It is a stated goal of the Spatial Information Services Stack (SISS) project that GeoServer community-schemas be ported to trunk. SISS is an NCRIS project http://www.ncris.dest.gov.au/ that will commence in July. It is anticipated that the port be completed this year. It is a goal of the SISS project that community-schemas become a fully-supported component of the GeoServer trunk. This cannot be achieved without the cooperation of the community.

Hi Ben; the community has a process that we use to run things (of which I am sure you are aware). So community involvement extends to voting on proposals you put forward, and if you are lucky testing by interested users.

I am helping a team right now do some work around WPS, and while it is our goal to have a module ready to go, we will need to go through the GSIP process and schedule any changes we make with other developers such as yourself.

The SISS project will have the same kind of relationship with GeoServer, ie as community members we both need to play nice with others and follow the process as outlined.

Dr Robert Woodcock, Director of AuScope Grid http://www.auscope.org/ and Project Manager of SISS, approached OpenPlans regarding this project, and was offered a subscription to GeoServer Enterprise. This was not what he wanted. The SISS project can contribute effort to the port of community-schemas to trunk as a contributor to an Open Source project, not as a commercial customer. SISS is all about improving the capability of Open Source software to deliver spatial services. This software can then be adopted by others. SISS is an enabling project, not one that intends to deliver any data itself.

Sounds great; I am really looking forward to seeing this work go main stream. This is a developer list - so we will focus on the programming (in the form of Jira issues) and a little bit of product planning (In the form of GSIP change proposals). Support contracts and training is available from a range of organizations on this list, but is a little bit off topic.

As of this morning, Robert is a subscriber to this list, and will no doubt be keenly following this discussion.

I will wait for Robert to introduce himself.

Jody

15

Hi Jody and everyone else,

Jody said:

I will wait for Robert to introduce himself.

I don't normally haunt developer e-mail lists, preferring that my team
members do, which I will quickly add include Rob Atkinson, Simon Cox and
Ben Caradoc-Davies whom you probably know so you have some idea of the
relationships.

I'd like to provide some information on activities with which I am
involved as I believe they can provide significant assistance to the
Geoserver (generally the open source geospatial) community whilst also
achieving the outcomes required by my stakeholders. I apologise in
advance as this will be a little long and may be off topic for a
developers list - I do however believe at least some is relevant and
important as there are significant software development resources
available.

The part I need some help with is working out how to interact with the
Geoserver community, of which "Porting community schemas to trunk" is
just one early component. By Australian standards the project I'm
involved with (AuScope & SISS) is large and involves open source
software and open standards along side industry, academia and government
use. I've no desire to disrupt the good work of the Geoserver community
by wandering in with a large project without due consideration of the
community interactions. Large projects can be both a blessing and a
curse - I'd like to avoid being a curse.

In short, I have $10 million between now and 2011 to establish a
community around a national spatial information infrastructure across
Australian Government, Industry and Research organisations based on open
source tools (or something like that). The nature of the funding is such
that strings are attached but it's surprisingly flexible and mostly
translates to a software development team for 3 years.

In many respects, the investment is intended to facilitate moving the
previous Australian testbed services community to be an Australian
production services community. In Australia at least, agencies are
struggling with how to work with open source and open standards and this
investment is helping to cross that hurdle.

On a more technical note, the Auscope & SISS infrastructure is currently
based on the following open source technologies:

* Geoserver - with community schema extensions as this is
considered essential

* GeoNetworks - with ANZLIC profiles

* Web Portals and Desktop clients - various samples are being
made available particularly for training and regression testing purposes
(e.g. Googlemap portal, udig, sample java desktop clients)

* OGC standards

* GeosciML standards for geoscience information

Whilst I believe the strategic intent of these activities, our
collaborations, and the investment level are capable of contributing to
the broadly desired outcomes, the move to production services and simply
having a large project does create some additional challenges both in
project coordination and the, more important, social interaction side of
the community.

Jody said:

The SISS project will have the same kind of relationship with

GeoServer, ie as community members we

both need to play nice with others and follow the process as outlined.

This is exactly what I'm wanting and clearly I need to become more
informed about the Geoserver community process. I'm concerned that these
mechanisms are intended for "smaller contributions" so to speak. All the
previous contributions to Geoserver I've heard of involve only a couple
of developers per feature. We could easily get to 8 or more. I'm not
certain the impact this effort would have on the rate of change and
administrative aspects of the Geoserver community. I also need to
understand how to have the Auscope+SISS development team work in its
"sandbox" whilst ensuring neither the Geoserver community nor the
Auscope+SISS developments hold each other up. The issue here isn't so
much developers but stakeholder management.

As Ben mentioned, I've had some non-list conversations with others in
the Geoserver community to try and investigate these issues but
responses so far (they are ongoing) aren't quite what I expected. Ben
already mentioned that some of the strings attached to the funding would
be a little strained if I contracted a commercial organisation that is
supporting the open source effort.

I have interested users and by Australian standards a lot of development
effort. I need to understand how AuScope & SISS can work with the open
source community - specifically Geoserver and Geotools.

Regards,

Rob

Dr Robert Woodcock

CSIRO Exploration & Mining ,

ARRC, 26 Dick Perry Ave,

Kensington, WA 6151 Australia

Phone +61 8 6436 8780 Fax +61 8 6436 8586

-----Original Message-----
From: geoserver-devel-bounces@lists.sourceforge.net
[mailto:geoserver-devel-bounces@lists.sourceforge.net] On Behalf Of Jody
Garnett
Sent: Wednesday, 21 May 2008 11:29 AM
To: Caradoc-Davies, Ben (E&M, Kensington)
Cc: Geoserver-devel
Subject: Re: [Geoserver-devel] Porting community schemas to trunk

Ben Caradoc-Davies wrote:

It is a stated goal of the Spatial Information Services Stack (SISS)

project that GeoServer community-schemas be ported to trunk. SISS is

an NCRIS project http://www.ncris.dest.gov.au/ that will commence in

July. It is anticipated that the port be completed this year. It is a

goal of the SISS project that community-schemas become a

fully-supported component of the GeoServer trunk. This cannot be

achieved without the cooperation of the community.

Hi Ben; the community has a process that we use to run things (of which

I am sure you are aware). So community involvement extends to voting on

proposals you put forward, and if you are lucky testing by interested
users.

I am helping a team right now do some work around WPS, and while it is

our goal to have a module ready to go, we will need to go through the

GSIP process and schedule any changes we make with other developers such

as yourself.

The SISS project will have the same kind of relationship with GeoServer,

ie as community members we both need to play nice with others and follow

the process as outlined.

Dr Robert Woodcock, Director of AuScope Grid http://www.auscope.org/

and Project Manager of SISS, approached OpenPlans regarding this

project, and was offered a subscription to GeoServer Enterprise. This

was not what he wanted. The SISS project can contribute effort to the

port of community-schemas to trunk as a contributor to an Open Source

project, not as a commercial customer. SISS is all about improving the

capability of Open Source software to deliver spatial services. This

software can then be adopted by others. SISS is an enabling project,

not one that intends to deliver any data itself.

Sounds great; I am really looking forward to seeing this work go main

stream. This is a developer list - so we will focus on the programming

(in the form of Jira issues) and a little bit of product planning (In

the form of GSIP change proposals). Support contracts and training is

available from a range of organizations on this list, but is a little

bit off topic.

As of this morning, Robert is a subscriber to this list, and will no

doubt be keenly following this discussion.

I will wait for Robert to introduce himself.

Jody

------------------------------------------------------------------------
-

This SF.net email is sponsored by: Microsoft

Defy all challenges. Microsoft(R) Visual Studio 2008.

http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________

Geoserver-devel mailing list

Geoserver-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geoserver-devel

16

Well me Rob, I am just going to get back to you with respect to "scope"..

This is exactly what I’m wanting and clearly I need to become more informed about the Geoserver community process. I’m concerned that these mechanisms are intended for “smaller contributions” so to speak. All the previous contributions to Geoserver I’ve heard of involve only a couple of developers per feature.

You will find a range of topics have been addressed that include the work of larger teams; the original user interface (and rewrite of the data access layer) was done with a team of 4 over the course of a year for example. The current migration to a new user interface will involve many developers (around June as I understand it), in terms of planning we just need to know what peoples delivery dates are and make sure the code base is stable when needed. I am carefully keeping the WPS project away from any kind of configuration / user interface concerns until after the work in June is complete for example.

We could easily get to 8 or more. I’m not certain the impact this effort would have on the rate of change and administrative aspects of the Geoserver community. I also need to understand how to have the Auscope+SISS development team work in its “sandbox” whilst ensuring neither the Geoserver community nor the Auscope+SISS developments hold each other up. The issue here isn’t so much developers but stakeholder management.

Understood; you are well represented by Rob Atkinson who is a member of our project steering committee. In general we ask that organizations that are involved heavily in GeoServer development and direction cough of someone to sit on our committee and take responsibility for the actions that are being taken :slight_smile:

As Ben mentioned, I’ve had some non-list conversations with others in the Geoserver community to try and investigate these issues but responses so far (they are ongoing) aren’t quite what I expected. Ben already mentioned that some of the strings attached to the funding would be a little strained if I contracted a commercial organisation that is supporting the open source effort.

We are suffering from two things at the moment:
- a very active development road map for the summer; it is hard for us to see past the current workload (that is what we have a steering committee to do after all)
- a very difficult problem with a poor record of communication; the "community schema" idea sounds simple on paper - but is far removed from the day-2-day experience of us here on the developer list. You are "blessed" with existing well defined application schemas that are both exacting and presumably helpful. This is so far away from what most of us get to see it is hard for us to understand it is a serious problem.

The larger problem appears to be behind us:
- we lacked a representation of Feature that was capable of addressing your concerns

I would like to confirm:
- is the "simple" flavour of Geometry supported by GeoTools right now (ie from SFSQL) is this sufficient for your needs?

I have interested users and by Australian standards a lot of development effort. I need to understand how AuScope & SISS can work with the open source community – specifically Geoserver and Geotools.

Both projects have taken steps in the last couple of years to answer your question:
- they feature a formal process that can be used to offer a direction to the community; with strict time limits for the community to respond (so your project is not hung while waiting for feedback), the usual open source controls are in place so -1 vote has a consequence in terms of helping you set up an alternative etc...
- they have form of community or unsupported space for experiments that are aligned with trunk to take place (so we don't end up with a situation like now where you are pouring money into a branch that will not see the light of day)

Well met,
Jody

17

robert.woodcock@anonymised.com ha scritto:
...

The SISS project will have the same kind of relationship with GeoServer,

ie as community members we

both need to play nice with others and follow the process as outlined.

This is exactly what I’m wanting and clearly I need to become more informed about the Geoserver community process. I’m concerned that these mechanisms are intended for “smaller contributions” so to speak. All the previous contributions to Geoserver I’ve heard of involve only a couple of developers per feature. We could easily get to 8 or more. I’m not certain the impact this effort would have on the rate of change and administrative aspects of the Geoserver community. I also need to understand how to have the Auscope+SISS development team work in its “sandbox” whilst ensuring neither the Geoserver community nor the Auscope+SISS developments hold each other up. The issue here isn’t so much developers but stakeholder management.

We have only a place for "smaller contributions" for a reason of self preservation. If you look at the GSIP process, which is used for non
trivial changes, you'll find out that no matter what, each of them
required a relatively small amount of changes to be performed.

The main reason behind that is that even with such smallish changes
the likeliness of breaking GeoServer stability is not trivial, so each
of them has been made in isolation, and with a small period of calm
in between them so that things were allowed to settle down and go back
to stability.

The way to introduce big changes without breaking the GeoServer stability, on which users and at least 3 businesses around depend,
is to slice change into manageable amount that get merged onto
the GeoServer trunk one at a time, each one with a separate
proposal and a review, in order to keep the code working, and every
developer on the same page.

Release wise this would work a lot better, since we're aiming at
making geoserver releases more frequent and with less big changes
per release, again, in an attempt to make GeoServer solid. This
affects more the stable series, but also means we're trying to
shorten the time between major ones, meaning that a "trunk" is
going to be switched over to a stable branch probably every 6 months
or so

A team of 8 people working full time is scary in this respect, because
it's going to produce big changes no matter what. Those 8 will sum
up to the already existing 7-10 developers working on GeoServer,
increasing a lot communication routes and generating serious develpment scalability issues. In order to avoid problems there, it is necessary
to split the work so that the areas of friction are minimal.

This is relatively easy with things like the WPS module RR is
developing, because it is separated from the other serfices, but not
that easy when we start talking about complex features, since that
encompasses the whole GeoServer/GeoTools architecture.

As Ben mentioned, I’ve had some non-list conversations with others in the Geoserver community to try and investigate these issues but responses so far (they are ongoing) aren’t quite what I expected. Ben already mentioned that some of the strings attached to the funding would be a little strained if I contracted a commercial organisation that is supporting the open source effort.

I think the proposal for a commercial support was made in good faith.
TOPP/OpenGeo is at the moment the major player in GeoServer development,
and it's getting quite a bit of contracts lately, meaning
we are having less and less time to dedicate to the pure open source
community support, and that even on the spot commercial requests
might have to be put in a queue.
A contract with OpenGeo would guarantee we're there when you need
to discuss further developments, or need a helping hand understanding
some of the gt/gs code, or reviewing and testing your changes and so on,
that is, it's just one way to make sure core developers are available.

We care about GeoServer, and we'll be there even without a contract,
it's just we cannot guarantee how prompt reactions will be, and how much time we'll be able to dedicate to it. The development you're proposing
is very interesting, I'd hate not being able to attend to its evolution/integration in a timely fashion, that's all.

Cheers
Andrea

18

Andrea Aime wrote:

This is relatively easy with things like the WPS module RR is
developing,

RR ?

--------------------
    Luca Morandini
www.lucamorandini.it
--------------------

19

Refractions Research (ie the company at the end of my email address). We have have a good working relationship with GeoServer as a project; and the Open Planning Project as an organization and have been able to field significant projects against the GeoServer code base.

Cheers,
Jody

Andrea Aime wrote:
  

This is relatively easy with things like the WPS module RR is
developing,
    
RR ?

--------------------
    Luca Morandini
www.lucamorandini.it
--------------------

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
  

20

Jody Garnett wrote:

Refractions Research (ie the company at the end of my email address). We have have a good working relationship with GeoServer as a project; and the Open Planning Project as an organization and have been able to field significant projects against the GeoServer code base.

Thanks for clarifying the issue :slight_smile:

Since we are at that: where's the source code of said WPS module ?

I was feeling like connecting it to GRASS via the JGRASS console engine... well at least trying it.

Regards,

--------------------
    Luca Morandini
www.lucamorandini.it
--------------------