···
On Wed, Nov 25, 2015 at 1:03 PM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:
Hi Adrea,
Do you have any comments on my email last week?
Sampo
–
On Thu, Nov 19, 2015 at 12:44 PM, Sampo Savolainen <sampo.savolainen@…3889…> wrote:
Hi,
Currently, GeoServer has the flag skipNumberMatched for FeatureTypeInfo, which can be used to disable calculating the number of matched results. So what I’m proposing is not at all new for GeoServer. However there are still some places (the ones I mentioned) where GetFeature might calculate the number of features even when this flag is enabled and the information is not necessary to enforce paging semantics. By my logic think we can safely skip counting features in those two places when skipNumberMatched is true and there is only one query in the GetFeature request (=because we don’t need to figure out which queries and offsets should be included in this page).
If you can confirm my hypothesis, this issue / patch can be treated as just an improvement to the existing skipNumberMatched feature that does not change query / paging semantics in any shape or form.
Sampo
–
Sampo Savolainen
Managing Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc
This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.
On Sun, Nov 15, 2015 at 11:20 AM, Andrea Aime <andrea.aime@anonymised.com.1268…> wrote:
Hi Sampo,
if I understand your proposal correctly, you’re suggesting to break both standard and backwards compatibility, which
is not something I’m too keen to accept :-p
Unless I’m misunderstanding… been reviewing/fixing/merging pull requests all weekend and I’m pretty tired by now
(anyone reading: really need some help there btw).
The “number matched” count is available on in WFS 2.0, it’s a second count that’s used to notify the total number
of matched features in a paging situation.
So… we cannot have it skip also normal feature counting, that has a different meaning.
I checked if there is a way not to report the feature count, stating the value
is “unknown”. As far as I can see, in WFS 1.1 the numberOfFeatures attribute is optional:
http://schemas.opengis.net/wfs/1.1.0/wfs.xsd
However, it seems that in wfs 2.0 there is no salvation, the local count is mandatory:
http://schemas.opengis.net/wfs/2.0/wfs.xsd
In particular:
<xsd:attribute name=“numberMatched” type=“wfs:nonNegativeIntegerOrUnknown” use=“required”/>
<xsd:attribute name=“numberReturned” type=“xsd:nonNegativeInteger” use=“required”/>
Another possible approach for very slow simple feature collections should be to dump
them on disk to avoid hitting the database again, it’s something we do in sorting/z-ordering in geotools for shapefiles
and friends, using a highly optimized binary feature serializer:
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/data/sort/SimpleFeatureIO.java
The sorting logic keeps a certain amount of features in memory (up to 1000 normally), past that they get dumped on disk
instead. Could be an option I guess, but yep, it would still require an extra flag in FeatureTypeInfo.
Cheers
Andrea
–
Sampo Savolainen
Managing Director, Spatineo Oy
sampo.savolainen@anonymised.com89…
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc
This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.
On Thu, Nov 12, 2015 at 3:31 PM, Sampo Savolainen <sampo.savolainen@anonymised.com> wrote:
Hi,
I recently created a ticket with a pull request to make GetFeature queries faster for data stores where counting features is slow. https://osgeo-org.atlassian.net/browse/GEOS-7301
In my pull request I proposed adding a new field to FeatureTypeInfo that would allow GeoServer to skip counting features altogether (under certain conditions) that would make things faster, but break strict compatibility with some types of queries. However I only later noted there is a flag already which does this in part: skipNumberMatched. I updated the ticket with a new proposal that I’ld like to get some comments on:
"I was looking at the existing flag in FeatureTypeInfo, skipNumberMatched. The implementation looks very close to what I’m trying to achieve here, but the current accepted implementation does not make GeoServer skip counting all features. More specifically, I’m referring to these rows of code:
https://github.com/geoserver/geoserver/blob/2e19a4ebe7f9c3698ac3eecf002da76bff0aa506/src/wfs/src/main/java/org/geoserver/wfs/GetFeature.java#L531
https://github.com/geoserver/geoserver/blob/2e19a4ebe7f9c3698ac3eecf002da76bff0aa506/src/wfs/src/main/java/org/geoserver/wfs/GetFeature.java#L552
Would you be willing to accept a patch where those counts are skipped if both:
-
primary feature of the query has skipNumberMatched = true, and
-
there is only a single query in the GetFeature request (paging
… ?
This would simplify this patch to only affecting GetFeature.java and remove the need for a new field in FeatureTypeInfo."
Can you think of any new problems with this approach (meaning that I should stick with adding a new flag to FeatureTypeInfo) or should I update my pull request based on this idea?
Sampo
–
Sampo Savolainen
Managing Director, Spatineo Oy
sampo.savolainen@anonymised.com
+358-407555649
Linnankoskenkatu 16 A 17, 00250 Helsinki, Finland
www.spatineo.com, twitter.com/#!/spatineo
www.linkedin.com/company/spatineo-inc
This message may contain privileged and/or confidential information. If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate, or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
–
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.