[Geoserver-users] Getting features intersecting with a polygon

Hi list,

We are using GeoServer 2.6.2 and I want to get all the features from a given layer (based on a SQL Server table) intersecting with a given polygon. So I created the following GET-request which returns a bunch of features (as expected).

geoserver/wfs?service=wfs&request=GetFeature&version=1.1.0&typeName=JAGIS:gemeente&outputFormat=GML2&cql_filter=INTERSECTS(SP_GEOMETRY, POLYGON ((142578.64599609 252217.79003906, 73781.897460938 141983.61767578, 287078.38037109 146764.85888672, 142578.64599609 252217.79003906)))

Because I don’t know how big the polygon is going to get, I want to send a POST-request, so I created (using the demo requests of geoserver) the wfs GetFeature-request (at the bottom of this message) which (in my humble opinion) should return the same bunch of features returned by the GET-request. But for some reason I can’t figure out the returned feature collection is empty. What am I missing here? If I check the geoserver logs, I see this request being logged. So for some reason an empty polygon is used.

Request: getFeature

service = WFS

version = 1.1.0

baseUrl = …

query[0]:

filter = [ SP_GEOMETRY intersects POLYGON EMPTY ]

typeName[0] = {JAGIS}gemeente

outputFormat = text/xml; subtype=gml/3.1.1

resultType = results

Kind regards,

Roel De Nijs

Senior Java Developer

<wfs:GetFeature service=“WFS” version=“1.1.0”

xmlns:wfs=“http://www.opengis.net/wfs

xmlns=“http://www.opengis.net/ogc

xmlns:gml=“http://www.opengis.net/gml

xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance

xsi:schemaLocation="http://www.opengis.net/wfs

http://schemas.opengis.net/wfs/1.1.0/wfs.xsd">

<wfs:Query typeName=“JAGIS:gemeente”>

SP_GEOMETRY

<gml:Polygon srsName=“http://www.opengis.net/gml/srs/epsg.xml#31370”>

gml:outerBoundaryIs

gml:LinearRing

<gml:coordinates decimal=“.” cs=“,” ts=" ">

142578.64599609,252217.79003906 73781.897460938,141983.61767578 287078.38037109,146764.85888672 142578.64599609,252217.79003906

</gml:coordinates>

</gml:LinearRing>

</gml:outerBoundaryIs>

</gml:Polygon>

</wfs:Query>

</wfs:GetFeature>

Hi list,

Seems I solved this issue myself. When I replace ‘outerBoundaryIs’ by ‘exterior’, the polygon isn’t empty anymore and the same bunch of features as with the GET-request are returned. Yay!

Now I have another question. When I turned verbose logging on I see the following query being executed fort this request:

SELECT “prop1”,“prop2” FROM “table1” WHERE “the_geom”.Filter(geometry::STGeomFromText(‘POLYGON ((142578.64599609 252217.79003906, 73781.897460938 141983.61767578, 287078.38037109 146764.85888672, 142578.64599609 252217.79003906))’, 31370)) = 1 AND “the_geom”.STIntersects(geometry::STGeomFromText(‘POLYGON ((142578.64599609 252217.79003906, 73781.897460938 141983.61767578, 287078.38037109 146764.85888672, 142578.64599609 252217.79003906))’, 31370)) = 1

Why does the where-clause has two conditions: Filter and STIntersects? I would only expecte the STIntersects one.

Kind regards,

Roel De Nijs

Senior Java Developer

···

Van: Roel De Nijs [mailto:roel.denijs@anonymised.com]
Verzonden: woensdag 18 maart 2015 11:50
Aan: Mailinglist: Geoserver - Users
Onderwerp: [Geoserver-users] Getting features intersecting with a polygon

Hi list,

We are using GeoServer 2.6.2 and I want to get all the features from a given layer (based on a SQL Server table) intersecting with a given polygon. So I created the following GET-request which returns a bunch of features (as expected).

geoserver/wfs?service=wfs&request=GetFeature&version=1.1.0&typeName=JAGIS:gemeente&outputFormat=GML2&cql_filter=INTERSECTS(SP_GEOMETRY, POLYGON ((142578.64599609 252217.79003906, 73781.897460938 141983.61767578, 287078.38037109 146764.85888672, 142578.64599609 252217.79003906)))

Because I don’t know how big the polygon is going to get, I want to send a POST-request, so I created (using the demo requests of geoserver) the wfs GetFeature-request (at the bottom of this message) which (in my humble opinion) should return the same bunch of features returned by the GET-request. But for some reason I can’t figure out the returned feature collection is empty. What am I missing here? If I check the geoserver logs, I see this request being logged. So for some reason an empty polygon is used.

Request: getFeature

service = WFS

version = 1.1.0

baseUrl = …

query[0]:

filter = [ SP_GEOMETRY intersects POLYGON EMPTY ]

typeName[0] = {JAGIS}gemeente

outputFormat = text/xml; subtype=gml/3.1.1

resultType = results

Kind regards,

Roel De Nijs

Senior Java Developer

<wfs:GetFeature service=“WFS” version=“1.1.0”

xmlns:wfs=“http://www.opengis.net/wfs

xmlns=“http://www.opengis.net/ogc

xmlns:gml=“http://www.opengis.net/gml

xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance

xsi:schemaLocation="http://www.opengis.net/wfs

http://schemas.opengis.net/wfs/1.1.0/wfs.xsd">

<wfs:Query typeName=“JAGIS:gemeente”>

SP_GEOMETRY

<gml:Polygon srsName=“http://www.opengis.net/gml/srs/epsg.xml#31370”>

gml:outerBoundaryIs

gml:LinearRing

<gml:coordinates decimal=“.” cs=“,” ts=" ">

142578.64599609,252217.79003906 73781.897460938,141983.61767578 287078.38037109,146764.85888672 142578.64599609,252217.79003906

</gml:coordinates>

</gml:LinearRing>

</gml:outerBoundaryIs>

</gml:Polygon>

</wfs:Query>

</wfs:GetFeature>


Volg Aquafin op Facebook | Twitter | YouTube | LinkedIN

Disclaimer: zie www.aquafin.be P Denk aan het milieu. Druk deze mail niet onnodig af.

On Wed, Mar 18, 2015 at 5:38 PM, Roel De Nijs <roel.denijs@anonymised.com>
wrote:

Hi list,

Seems I solved this issue myself. When I replace 'outerBoundaryIs' by
'exterior', the polygon isn’t empty anymore and the same bunch of features
as with the GET-request are returned. Yay!

Yep, the original GML was invalid, mixing GML 2 and GML3 elements.

Now I have another question. When I turned verbose logging on I see the
following query being executed fort this request:

SELECT "prop1","prop2" FROM "table1" WHERE
"the_geom".Filter(geometry::STGeomFromText('POLYGON ((142578.64599609
252217.79003906, 73781.897460938 141983.61767578, 287078.38037109
146764.85888672, 142578.64599609 252217.79003906))', 31370)) = 1 AND
"the_geom".STIntersects(geometry::STGeomFromText('POLYGON ((142578.64599609
252217.79003906, 73781.897460938 141983.61767578, 287078.38037109
146764.85888672, 142578.64599609 252217.79003906))', 31370)) = 1

Why does the where-clause has two conditions: Filter and STIntersects? I
would only expecte the STIntersects one.

Primary and secondary filter... the first one uses the spatial index, the
second does the in memory filtering.
Or has SQL Server learned to use the spatial index when just seeing
STIntersects?

Even if it does, we'd might have to keep the current syntax for backwards
compatibility with older versions

That said, SQLServer optimizer is really poor and expects some complicated
setup in order to better optimize the queries (basically, writing the query
in more than one bit, setting variables first in a few statements, then
using them in the query) which is difficult to support with our current
shared codebase for all databases... it's not impossible though, we are
basically waiting for people to sponsor this work, as it's not small (won't
fit in anybody's spare time).

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

geowolf wrote

On Wed, Mar 18, 2015 at 5:38 PM, Roel De Nijs &lt;

roel.denijs@

&gt;
wrote:

Primary and secondary filter... the first one uses the spatial index, the
second does the in memory filtering.
Or has SQL Server learned to use the spatial index when just seeing
STIntersects?

Even if it does, we'd might have to keep the current syntax for backwards
compatibility with older versions

That said, SQLServer optimizer is really poor and expects some complicated
setup in order to better optimize the queries (basically, writing the
query
in more than one bit, setting variables first in a few statements, then
using them in the query) which is difficult to support with our current
shared codebase for all databases... it's not impossible though, we are
basically waiting for people to sponsor this work, as it's not small
(won't
fit in anybody's spare time).

We use a few custom-written queries which use only an STIntersects in the
where-clause and the spatial index is mentioned in the execution plan of
this query. So I think SQL Server has learned to use the spatial index :slight_smile:
According to this blog post [1] it's very important to upgrade (if possible)
to SQL Server 2008 SP1 if you want to ensure the spatial index is being
used. (We use SQL Server 2008 R2)

I did a very simple test using one of our queries with different
combinations of Filter and/or STIntersects in the where-clause:
a) only Filter: 76159 records in 9032ms
b) only STIntersects: 67490 records in 11472ms
c) Filter and STIntersects: 67490 records in 21472ms
I executed every query using the SQL Server Management Studio and before
each query I removed the query plans.

I tried the 3rd point from the blog post [1] as well, but no significant
changes in query execution time, nor in query execution plans.

[1]
http://www.sqlskills.com/blogs/bobb/how-to-ensure-your-spatial-index-is-being-used/

Kind regards,
Roel De Nijs
Senior Java Developer

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Getting-features-intersecting-with-a-polygon-tp5194135p5195806.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

On Fri, Mar 27, 2015 at 7:36 PM, Roel De Nijs <roel.denijs@anonymised.com>
wrote:

We use a few custom-written queries which use only an STIntersects in the
where-clause and the spatial index is mentioned in the execution plan of
this query. So I think SQL Server has learned to use the spatial index :slight_smile:
According to this blog post [1] it's very important to upgrade (if
possible)
to SQL Server 2008 SP1 if you want to ensure the spatial index is being
used. (We use SQL Server 2008 R2)

Nah, we had no ends of problems with sql server 2012, that's why we got the
sponsoring
to add the query hints in just this database.
But it's nice that at least with simple queries, no parallel execution
options on,
the optimizer can do its job as it should (I guess we just got spoiled by
postgresql/postgis, there are no query hints, and at least in the last 8
years,
I had no need to use them anyways, Oracle and SQLServer are disappointing
by comparison, but not "bad" in absolute terms).

I did a very simple test using one of our queries with different
combinations of Filter and/or STIntersects in the where-clause:
a) only Filter: 76159 records in 9032ms
b) only STIntersects: 67490 records in 11472ms
c) Filter and STIntersects: 67490 records in 21472ms

Interesting, that's good data. Could you open a ticket asking for an
improvement (mind,
tomorrow we switch to a new jira server, so it's either tonight, or next
week).
We need to determine if the above are valid on all versions, or else, need
to setup
some version recognition and act accordingly

I executed every query using the SQL Server Management Studio and before
each query I removed the query plans.

Mind, the query performance from native sql server tools, and from jdbc,
sometimes
vary a lot.

I tried the 3rd point from the blog post [1] as well, but no significant
changes in query execution time, nor in query execution plans.

[1]

http://www.sqlskills.com/blogs/bobb/how-to-ensure-your-spatial-index-is-being-used/

Ok, this is a where postgis is vastly superior to both oracle and sqlserver.
In postgis you send a query with a bbox the bbox is parsed before the query
planner
runs, and it can intelligently decide whether to use the index or a
sequential scan.
The threshold to make index scans faster is surprisingly low, something
like extracting
5 to 7% of your data, this is because you have to jump a lot back and forth
in the
index to collect the record ids, causing quite a bit of extra wait on
spinnnig disk storage.

In SQL Server and Oracle, as far as I know, the query is planned before the
functions
creating the reference geometries are called, so the optimizer normally
decides to use
the spatial index (minus some patological cases, e.g., parallel query
execution) and
that's just wrong, performance wise, when you
are trying to access a significant portion of your data (which in mapping,
is not at all uncommon).

Soo.. in order to give the optimizer a chance to figure out how much data
is actually
going to be read, we should go though that @declare business... and this is
where we'd
need funding.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Finally after 6 months I have had some time to investigate this further. Of course I didn’t know the polygon anymore from my initial simple test, so I had to use another one. We are still using SQL Server 2008 R2.

I did a very simple test using one of our queries with different combinations of Filter and/or STIntersects in the where-clause:

a) only Filter: 80836 records in 4215ms

b) only STIntersects: 77729 records in 5017ms

c) Filter and STIntersects: 77729 records in 11364ms

I executed every query using the SQL Server Management Studio and before each query I removed the query plans.

Then I executed every query again using Microsoft JDBC Driver for SQL Server (version 4.0.2206.100) and the time needed to execute the query is pretty similar. Before executing each query, I removed the query plans as well.

a) only Filter: 80836 records in 4459ms

b) only STIntersects: 77729 records in 5180ms

c) Filter and STIntersects: 77729 records in 11604ms

If it’s still useful, I will open a jira ticket in the next few days and share these results.

Kind regards,

Roel De Nijs

Senior Java Developer

Verzonden: vrijdag 27 maart 2015 21:40

···

On Fri, Mar 27, 2015 at 7:36 PM, Roel De Nijs <roel.denijs@…5173…> wrote:

We use a few custom-written queries which use only an STIntersects in the
where-clause and the spatial index is mentioned in the execution plan of
this query. So I think SQL Server has learned to use the spatial index :slight_smile:
According to this blog post [1] it’s very important to upgrade (if possible)
to SQL Server 2008 SP1 if you want to ensure the spatial index is being
used. (We use SQL Server 2008 R2)

Nah, we had no ends of problems with sql server 2012, that’s why we got the sponsoring

to add the query hints in just this database.

But it’s nice that at least with simple queries, no parallel execution options on,

the optimizer can do its job as it should (I guess we just got spoiled by

postgresql/postgis, there are no query hints, and at least in the last 8 years,

I had no need to use them anyways, Oracle and SQLServer are disappointing

by comparison, but not “bad” in absolute terms).

I did a very simple test using one of our queries with different
combinations of Filter and/or STIntersects in the where-clause:
a) only Filter: 76159 records in 9032ms
b) only STIntersects: 67490 records in 11472ms
c) Filter and STIntersects: 67490 records in 21472ms

Interesting, that’s good data. Could you open a ticket asking for an improvement (mind,

tomorrow we switch to a new jira server, so it’s either tonight, or next week).

We need to determine if the above are valid on all versions, or else, need to setup

some version recognition and act accordingly

I executed every query using the SQL Server Management Studio and before
each query I removed the query plans.

Mind, the query performance from native sql server tools, and from jdbc, sometimes

vary a lot.

I tried the 3rd point from the blog post [1] as well, but no significant
changes in query execution time, nor in query execution plans.

[1]
http://www.sqlskills.com/blogs/bobb/how-to-ensure-your-spatial-index-is-being-used/

Ok, this is a where postgis is vastly superior to both oracle and sqlserver.

In postgis you send a query with a bbox the bbox is parsed before the query planner

runs, and it can intelligently decide whether to use the index or a sequential scan.

The threshold to make index scans faster is surprisingly low, something like extracting

5 to 7% of your data, this is because you have to jump a lot back and forth in the

index to collect the record ids, causing quite a bit of extra wait on spinnnig disk storage.

In SQL Server and Oracle, as far as I know, the query is planned before the functions

creating the reference geometries are called, so the optimizer normally decides to use

the spatial index (minus some patological cases, e.g., parallel query execution) and

that’s just wrong, performance wise, when you

are trying to access a significant portion of your data (which in mapping, is not at all uncommon).

Soo… in order to give the optimizer a chance to figure out how much data is actually

going to be read, we should go though that @declare business… and this is where we’d

need funding.

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi Roel,
yes, adding a ticket would not hurt. Is there anybody around with a more recent SQL server release that can perform
similar tests? It would be great to know if this finding is valid cross version

Cheers
Andrea

···

On Mon, Sep 28, 2015 at 5:59 PM, Roel De Nijs <roel.denijs@anonymised.com> wrote:

Finally after 6 months I have had some time to investigate this further. Of course I didn’t know the polygon anymore from my initial simple test, so I had to use another one. We are still using SQL Server 2008 R2.

I did a very simple test using one of our queries with different combinations of Filter and/or STIntersects in the where-clause:

a) only Filter: 80836 records in 4215ms

b) only STIntersects: 77729 records in 5017ms

c) Filter and STIntersects: 77729 records in 11364ms

I executed every query using the SQL Server Management Studio and before each query I removed the query plans.

Then I executed every query again using Microsoft JDBC Driver for SQL Server (version 4.0.2206.100) and the time needed to execute the query is pretty similar. Before executing each query, I removed the query plans as well.

a) only Filter: 80836 records in 4459ms

b) only STIntersects: 77729 records in 5180ms

c) Filter and STIntersects: 77729 records in 11604ms

If it’s still useful, I will open a jira ticket in the next few days and share these results.

Kind regards,

Roel De Nijs

Senior Java Developer

Van: andrea.aime@anonymised.com… [mailto:andrea.aime@anonymised.com] Namens Andrea Aime
Verzonden: vrijdag 27 maart 2015 21:40
Aan: Roel De Nijs <roel.denijs@anonymised.com>
CC: GeoServer Mailing List List <geoserver-users@anonymised.come.net>
Onderwerp: Re: [Geoserver-users] Getting features intersecting with a polygon

On Fri, Mar 27, 2015 at 7:36 PM, Roel De Nijs <roel.denijs@anonymised.com73…> wrote:

We use a few custom-written queries which use only an STIntersects in the
where-clause and the spatial index is mentioned in the execution plan of
this query. So I think SQL Server has learned to use the spatial index :slight_smile:
According to this blog post [1] it’s very important to upgrade (if possible)
to SQL Server 2008 SP1 if you want to ensure the spatial index is being
used. (We use SQL Server 2008 R2)

Nah, we had no ends of problems with sql server 2012, that’s why we got the sponsoring

to add the query hints in just this database.

But it’s nice that at least with simple queries, no parallel execution options on,

the optimizer can do its job as it should (I guess we just got spoiled by

postgresql/postgis, there are no query hints, and at least in the last 8 years,

I had no need to use them anyways, Oracle and SQLServer are disappointing

by comparison, but not “bad” in absolute terms).

I did a very simple test using one of our queries with different
combinations of Filter and/or STIntersects in the where-clause:
a) only Filter: 76159 records in 9032ms
b) only STIntersects: 67490 records in 11472ms
c) Filter and STIntersects: 67490 records in 21472ms

Interesting, that’s good data. Could you open a ticket asking for an improvement (mind,

tomorrow we switch to a new jira server, so it’s either tonight, or next week).

We need to determine if the above are valid on all versions, or else, need to setup

some version recognition and act accordingly

I executed every query using the SQL Server Management Studio and before
each query I removed the query plans.

Mind, the query performance from native sql server tools, and from jdbc, sometimes

vary a lot.

I tried the 3rd point from the blog post [1] as well, but no significant
changes in query execution time, nor in query execution plans.

[1]
http://www.sqlskills.com/blogs/bobb/how-to-ensure-your-spatial-index-is-being-used/

Ok, this is a where postgis is vastly superior to both oracle and sqlserver.

In postgis you send a query with a bbox the bbox is parsed before the query planner

runs, and it can intelligently decide whether to use the index or a sequential scan.

The threshold to make index scans faster is surprisingly low, something like extracting

5 to 7% of your data, this is because you have to jump a lot back and forth in the

index to collect the record ids, causing quite a bit of extra wait on spinnnig disk storage.

In SQL Server and Oracle, as far as I know, the query is planned before the functions

creating the reference geometries are called, so the optimizer normally decides to use

the spatial index (minus some patological cases, e.g., parallel query execution) and

that’s just wrong, performance wise, when you

are trying to access a significant portion of your data (which in mapping, is not at all uncommon).

Soo… in order to give the optimizer a chance to figure out how much data is actually

going to be read, we should go though that @declare business… and this is where we’d

need funding.

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



Volg Aquafin op Facebook | Twitter | YouTube | LinkedIN

Disclaimer: zie www.aquafin.be P Denk aan het milieu. Druk deze mail niet onnodig af.

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


I added a ticket: https://osgeo-org.atlassian.net/browse/GEOT-5236

If additional information is needed, just let me know and I’ll add a comment with the requested information.

Verzonden: dinsdag 29 september 2015 8:57

···

Hi Roel,

yes, adding a ticket would not hurt. Is there anybody around with a more recent SQL server release that can perform

similar tests? It would be great to know if this finding is valid cross version

Cheers

Andrea

On Mon, Sep 28, 2015 at 5:59 PM, Roel De Nijs <roel.denijs@…5173…> wrote:

Finally after 6 months I have had some time to investigate this further. Of course I didn’t know the polygon anymore from my initial simple test, so I had to use another one. We are still using SQL Server 2008 R2.

I did a very simple test using one of our queries with different combinations of Filter and/or STIntersects in the where-clause:

a) only Filter: 80836 records in 4215ms

b) only STIntersects: 77729 records in 5017ms

c) Filter and STIntersects: 77729 records in 11364ms

I executed every query using the SQL Server Management Studio and before each query I removed the query plans.

Then I executed every query again using Microsoft JDBC Driver for SQL Server (version 4.0.2206.100) and the time needed to execute the query is pretty similar. Before executing each query, I removed the query plans as well.

a) only Filter: 80836 records in 4459ms

b) only STIntersects: 77729 records in 5180ms

c) Filter and STIntersects: 77729 records in 11604ms

If it’s still useful, I will open a jira ticket in the next few days and share these results.

Kind regards,

Roel De Nijs

Senior Java Developer

Van: andrea.aime@…84… [mailto:andrea.aime@…84…] Namens Andrea Aime
Verzonden: vrijdag 27 maart 2015 21:40
Aan: Roel De Nijs <roel.denijs@…5173…>
CC: GeoServer Mailing List List <geoserver-users@lists.sourceforge.net>
Onderwerp: Re: [Geoserver-users] Getting features intersecting with a polygon

On Fri, Mar 27, 2015 at 7:36 PM, Roel De Nijs <roel.denijs@…5173…> wrote:

We use a few custom-written queries which use only an STIntersects in the
where-clause and the spatial index is mentioned in the execution plan of
this query. So I think SQL Server has learned to use the spatial index :slight_smile:
According to this blog post [1] it’s very important to upgrade (if possible)
to SQL Server 2008 SP1 if you want to ensure the spatial index is being
used. (We use SQL Server 2008 R2)

Nah, we had no ends of problems with sql server 2012, that’s why we got the sponsoring

to add the query hints in just this database.

But it’s nice that at least with simple queries, no parallel execution options on,

the optimizer can do its job as it should (I guess we just got spoiled by

postgresql/postgis, there are no query hints, and at least in the last 8 years,

I had no need to use them anyways, Oracle and SQLServer are disappointing

by comparison, but not “bad” in absolute terms).

I did a very simple test using one of our queries with different
combinations of Filter and/or STIntersects in the where-clause:
a) only Filter: 76159 records in 9032ms
b) only STIntersects: 67490 records in 11472ms
c) Filter and STIntersects: 67490 records in 21472ms

Interesting, that’s good data. Could you open a ticket asking for an improvement (mind,

tomorrow we switch to a new jira server, so it’s either tonight, or next week).

We need to determine if the above are valid on all versions, or else, need to setup

some version recognition and act accordingly

I executed every query using the SQL Server Management Studio and before
each query I removed the query plans.

Mind, the query performance from native sql server tools, and from jdbc, sometimes

vary a lot.

I tried the 3rd point from the blog post [1] as well, but no significant
changes in query execution time, nor in query execution plans.

[1]
http://www.sqlskills.com/blogs/bobb/how-to-ensure-your-spatial-index-is-being-used/

Ok, this is a where postgis is vastly superior to both oracle and sqlserver.

In postgis you send a query with a bbox the bbox is parsed before the query planner

runs, and it can intelligently decide whether to use the index or a sequential scan.

The threshold to make index scans faster is surprisingly low, something like extracting

5 to 7% of your data, this is because you have to jump a lot back and forth in the

index to collect the record ids, causing quite a bit of extra wait on spinnnig disk storage.

In SQL Server and Oracle, as far as I know, the query is planned before the functions

creating the reference geometries are called, so the optimizer normally decides to use

the spatial index (minus some patological cases, e.g., parallel query execution) and

that’s just wrong, performance wise, when you

are trying to access a significant portion of your data (which in mapping, is not at all uncommon).

Soo… in order to give the optimizer a chance to figure out how much data is actually

going to be read, we should go though that @declare business… and this is where we’d

need funding.

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.



Volg Aquafin op Facebook | Twitter | YouTube | LinkedIN

Disclaimer: zie www.aquafin.be P Denk aan het milieu. Druk deze mail niet onnodig af.

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Fri, Oct 2, 2015 at 3:36 PM, Roel De Nijs <roel.denijs@anonymised.com> wrote:

I added a ticket: https://osgeo-org.atlassian.net/browse/GEOT-5236

If additional information is needed, just let me know and I’ll add a
comment with the requested information.

I believe the ticket is good, thank you. Now we just need someone that will
work on it (finding someone that decides to
spend his/her spare time improving support for a commercial dbms is...
uh... not impossible but somewhat unlikely)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------