[Geoserver-devel] WFS paging on 2.1.x

GeoServer GeoServer Developer
1

Hi all,
as some of you might now I had the get go to work on WFS paging support
on the 2.1.x series.

The work is based on Justins’s patch here http://jira.codehaus.org/browse/GEOS-4485,
work that has been checked for compatibilty with whatever we have on trunk
and augmented to address concerns raised when Justin last proposed it on
the maling list.

The main concern expressed last time was that only a few selected stores would
support paging, in particular only JDBC based ones, but not for example shapefiles.
In the meantime a generic memory/disk based feature sorter has been developed
on trunk that allows to support sorting and offsets (both required for paging)
on top of a generic feature reader/feature iterator.

The improvement made on top of the old patch was to have GeoServer own
FeatureSource wrapper use the above sorter, backported from geotools trunk
to 2.7.x, to fully support both sorting and paging on all data store that
cannot do the service by themselves.
This is actually more general than what we have on trunk, so I was thinking
to forward port this change there too.

The original paging tests have been replaced with ones ported back from
trunk and they keep on working.

After the fact I’ve noticed that on trunk a similar approach was taken, but
only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?
In my patch when a start offset has been specified natural sorting is imposed
even if not asked directly.

Anyways, a bit more work seems to be needed in order to harmonize the
two branches but things appear to be looking good.

I’ve attached the two updated patches on the jira, look for the AA suffix:
http://jira.codehaus.org/browse/GEOS-4485

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

2

After the fact I’ve noticed that on trunk a similar approach was taken, but

only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?

I am using the paging, with the “natural order” of the dataset; to great effect currently (on a uDig based project). I hope to be able to back port the changes as they are very impressive.

Jody

3

On Sun, Dec 11, 2011 at 3:03 PM, Jody Garnett <jody.garnett@…403…> wrote:

After the fact I’ve noticed that on trunk a similar approach was taken, but

only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?

I am using the paging, with the “natural order” of the dataset; to great effect currently (on a uDig based project). I hope to be able to back port the changes as they are very impressive.

Indeed I’m also using natural order in my patch, but GeoServer trunk does not do that, hence me asking.
If you look at my GeoTools patch you’ll also notice that I’ve modified the shapefile data store to advertise
that the natural ordering is supported, since no matter what we always return data in the fid order
(I also checked, there is no other ordering support around in the normal shapefile reader or in its base
classes, none that I can see at least).

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

4

On Fri, Dec 9, 2011 at 1:51 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi all,
as some of you might now I had the get go to work on WFS paging support
on the 2.1.x series.

The work is based on Justins’s patch here http://jira.codehaus.org/browse/GEOS-4485,
work that has been checked for compatibilty with whatever we have on trunk
and augmented to address concerns raised when Justin last proposed it on
the maling list.

The main concern expressed last time was that only a few selected stores would
support paging, in particular only JDBC based ones, but not for example shapefiles.
In the meantime a generic memory/disk based feature sorter has been developed
on trunk that allows to support sorting and offsets (both required for paging)
on top of a generic feature reader/feature iterator.

The improvement made on top of the old patch was to have GeoServer own
FeatureSource wrapper use the above sorter, backported from geotools trunk
to 2.7.x, to fully support both sorting and paging on all data store that
cannot do the service by themselves.
This is actually more general than what we have on trunk, so I was thinking
to forward port this change there too.

The original paging tests have been replaced with ones ported back from
trunk and they keep on working.

After the fact I’ve noticed that on trunk a similar approach was taken, but
only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?
In my patch when a start offset has been specified natural sorting is imposed
even if not asked directly.

We went around on this a few times but it was thought that it would be more desirable to simply do the offset regardless of whether the underlying format could sort, rather than throw an exception back. Accepting the fact that using a non sorting store will lead to inconsistent results. This also seemed to fit with how we handle transactions… allowing them on stores that can actually can’t really do transactions, rather than throwing an exception back. Not a perfect analogy I know. But the fact that the mergesort feature stuff was coming down the pipe and we would eventually be able to supplement the non sorting stores regardless it seemed acceptable.

Anyways, a bit more work seems to be needed in order to harmonize the
two branches but things appear to be looking good.

I’ve attached the two updated patches on the jira, look for the AA suffix:
http://jira.codehaus.org/browse/GEOS-4485

Looked over the patches and they look good to me. Do we have a handle on the issues merging with trunk?

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

Cloud Services Checklist: Pricing and Packaging Optimization
This white paper is intended to serve as a reference, checklist and point of
discussion for anyone considering optimizing the pricing and packaging model
of a cloud services business. Read Now!
http://www.accelacomm.com/jaw/sfnl/114/51491232/

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

On Sun, Dec 11, 2011 at 6:50 PM, Justin Deoliveira <jdeolive@anonymised.com.1501…> wrote:

After the fact I’ve noticed that on trunk a similar approach was taken, but
only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?
In my patch when a start offset has been specified natural sorting is imposed
even if not asked directly.

We went around on this a few times but it was thought that it would be more desirable to simply do the offset regardless of whether the underlying format could sort, rather than throw an exception back. Accepting the fact that using a non sorting store will lead to inconsistent results. This also seemed to fit with how we handle transactions… allowing them on stores that can actually can’t really do transactions, rather than throwing an exception back. Not a perfect analogy I know. But the fact that the mergesort feature stuff was coming down the pipe and we would eventually be able to supplement the non sorting stores regardless it seemed acceptable.

I see. However with the patch I’ve made we can basically add sorting support where
none is found.
There are two sides indeed:
a) add sorting support for all, regardless of the native support
b) force natural sorting during paging when no other sorting is around

a) seems to be nice in general, along with the lines that WFS hides whatever
capabilities the actual underlying data storage has.
b) is doable, but takes a toll on paging if the native store does not support
natural sorting

At the very least I’d like to support a), b) I don’t really have an opinion on.

Anyways, a bit more work seems to be needed in order to harmonize the
two branches but things appear to be looking good.

I’ve attached the two updated patches on the jira, look for the AA suffix:
http://jira.codehaus.org/browse/GEOS-4485

Looked over the patches and they look good to me. Do we have a handle on the issues merging with trunk?

Afaik there is not much to merge on trunk, besides that GeoServerFeatureCollection.
If we go for just supporting a) I guess I could still use GeoServerFeatureCollection,
just making sure it simply does skip and max, or roll a new SortinFeatureCollection
that only does sorting and keep MaxFeatureCollection (which would have to be
back-ported to 2.1.x).

Regarless the paging tests would be exactly the same as on trunk.
I did not touch much else on the original patch, do you know if there are other
significant differences?
Like, in the bindings for example?

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

6

On Sun, Dec 11, 2011 at 8:32 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Dec 11, 2011 at 6:50 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

After the fact I’ve noticed that on trunk a similar approach was taken, but
only to support offset without caring for sorting. I thought we wanted to have paging
only when stabilized by a sort, am I mistaken?
In my patch when a start offset has been specified natural sorting is imposed
even if not asked directly.

We went around on this a few times but it was thought that it would be more desirable to simply do the offset regardless of whether the underlying format could sort, rather than throw an exception back. Accepting the fact that using a non sorting store will lead to inconsistent results. This also seemed to fit with how we handle transactions… allowing them on stores that can actually can’t really do transactions, rather than throwing an exception back. Not a perfect analogy I know. But the fact that the mergesort feature stuff was coming down the pipe and we would eventually be able to supplement the non sorting stores regardless it seemed acceptable.

I see. However with the patch I’ve made we can basically add sorting support where
none is found.
There are two sides indeed:
a) add sorting support for all, regardless of the native support
b) force natural sorting during paging when no other sorting is around

a) seems to be nice in general, along with the lines that WFS hides whatever
capabilities the actual underlying data storage has.
b) is doable, but takes a toll on paging if the native store does not support
natural sorting

At the very least I’d like to support a), b) I don’t really have an opinion on.

Right… makes sense. I would say only doing (a) is fine for now and just continue to do what we do now and just ignore the fact that there is no sorting, and page regardless.

Anyways, a bit more work seems to be needed in order to harmonize the
two branches but things appear to be looking good.

I’ve attached the two updated patches on the jira, look for the AA suffix:
http://jira.codehaus.org/browse/GEOS-4485

Looked over the patches and they look good to me. Do we have a handle on the issues merging with trunk?

Afaik there is not much to merge on trunk, besides that GeoServerFeatureCollection.
If we go for just supporting a) I guess I could still use GeoServerFeatureCollection,
just making sure it simply does skip and max, or roll a new SortinFeatureCollection
that only does sorting and keep MaxFeatureCollection (which would have to be
back-ported to 2.1.x).

cool

Regarless the paging tests would be exactly the same as on trunk.
I did not touch much else on the original patch, do you know if there are other
significant differences?
Like, in the bindings for example?

hmmm… none I can think of at the moment… i don’t expect any of parsing code, etc… will have to change.

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

7

On Mon, Dec 12, 2011 at 3:02 PM, Justin Deoliveira <jdeolive@anonymised.com.1501…> wrote:

There are two sides indeed:
a) add sorting support for all, regardless of the native support
b) force natural sorting during paging when no other sorting is around

a) seems to be nice in general, along with the lines that WFS hides whatever
capabilities the actual underlying data storage has.
b) is doable, but takes a toll on paging if the native store does not support
natural sorting

At the very least I’d like to support a), b) I don’t really have an opinion on.

Right… makes sense. I would say only doing (a) is fine for now and just continue to do what we do now and just ignore the fact that there is no sorting, and page regardless.

Cool, works for me.

Afaik there is not much to merge on trunk, besides that GeoServerFeatureCollection.
If we go for just supporting a) I guess I could still use GeoServerFeatureCollection,
just making sure it simply does skip and max, or roll a new SortinFeatureCollection
that only does sorting and keep MaxFeatureCollection (which would have to be
back-ported to 2.1.x).

cool

Ah hem… which one is cooler? :slight_smile:
Any preference between:

  • still use GeoServerFeatureCollection, just making sure it simply does skip and max
    (which would be forward ported to trunk and replace MaxFeatureCollection)
  • roll a new SortingFeatureCollection that only does sorting and keep
    MaxFeatureCollection (which would have to be back-ported to 2.1.x).

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

8

On Mon, Dec 12, 2011 at 2:22 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Mon, Dec 12, 2011 at 3:02 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

There are two sides indeed:
a) add sorting support for all, regardless of the native support
b) force natural sorting during paging when no other sorting is around

a) seems to be nice in general, along with the lines that WFS hides whatever
capabilities the actual underlying data storage has.
b) is doable, but takes a toll on paging if the native store does not support
natural sorting

At the very least I’d like to support a), b) I don’t really have an opinion on.

Right… makes sense. I would say only doing (a) is fine for now and just continue to do what we do now and just ignore the fact that there is no sorting, and page regardless.

Cool, works for me.

Afaik there is not much to merge on trunk, besides that GeoServerFeatureCollection.
If we go for just supporting a) I guess I could still use GeoServerFeatureCollection,
just making sure it simply does skip and max, or roll a new SortinFeatureCollection
that only does sorting and keep MaxFeatureCollection (which would have to be
back-ported to 2.1.x).

cool

Ah hem… which one is cooler? :slight_smile:
Any preference between:

  • still use GeoServerFeatureCollection, just making sure it simply does skip and max
    (which would be forward ported to trunk and replace MaxFeatureCollection)
  • roll a new SortingFeatureCollection that only does sorting and keep

MaxFeatureCollection (which would have to be back-ported to 2.1.x).

Oops, sorry :slight_smile: … no strong preference but if I had to choose I would go with the second, seems a bit cleaner to keep them separate but like i said no real opinion, if the implementation of the first turns out to be easier i am all for that.

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

–
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

9

On Mon, Dec 12, 2011 at 3:27 PM, Justin Deoliveira <jdeolive@anonymised.com.1501…> wrote:

Ah hem… which one is cooler? :slight_smile:

Any preference between:

  • still use GeoServerFeatureCollection, just making sure it simply does skip and max
    (which would be forward ported to trunk and replace MaxFeatureCollection)
  • roll a new SortingFeatureCollection that only does sorting and keep

MaxFeatureCollection (which would have to be back-ported to 2.1.x).

Oops, sorry :slight_smile: … no strong preference but if I had to choose I would go with the second, seems a bit cleaner to keep them separate but like i said no real opinion, if the implementation of the first turns out to be easier i am all for that.

No problemo, I’ll go for the second :slight_smile:

Cheers
Andrea

–

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf