[Geoserver-devel] CQL filter OR operator does not work on attributes that are multiple on MongoDB DataSource

Artem_Groznykh · June 20, 2019, 4:54pm

Hi all

I use GeoServer 2.15 and MongoDB 3.6

I've created a collection with data from here
<https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html>
(just two records), created 2dsphere index

Then I created a layer from that collection and tried to request it by WMS
with cql_filter in URL params.

1. when I use simple attributes (that are not multiple) in the filter all is
fine (points are visible):

- filter: "contract.mail" ILIKE '%@mail.com%' OR "name" ILIKE '%station%'

- mongo log: { ... , filter: { }, projection: { geometry: 1,
contact.mail: 1, name: 1 }, ... }

<http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png>

2. when I use attributes that are multiple in the filter then it DOES NOT
work (there is no any point on received PNG):

- filter: "measurements.name" ILIKE '%wind%' OR "measurements.unit" ILIKE
'%c%'

- mongo log: { ... , filter: { }, projection: {geometry: 1,
measurements.unit: 1, measurements.name: 1}, ... }

<http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(1).png>

3. when I change OR to AND in previous filter then it's works fine:

- filter: "measurements.name" ILIKE '%wind%' AND "measurements.unit" ILIKE
'%c%'

- mongo log: { ..., filter: { $and: [ { measurements.name: /^.*wind.*$/i
}, { measurements.unit: /^.*p.*$/i } ] }, projection: { geometry: 1 }, ... }

<http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(2).png>

So I concluded that GeoServer does AND operation by sending filter to
mongoDB. And OR operation by itself requesting all records from collection.
But with attributes that are multiple something goes wrong. I think it's a
bug.

If you have any sugestions please tell me.

--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html

Nuno_Oliveira1 · June 21, 2019, 8:51am

Hi,
thank you for the detailed feedback, indeed at first look this looks like a bug to me.

Can you create Jira ticket for this? Instructions:
http://geoserver.org/issues/

Best regards,
Nuno Oliveira

On Thu, 2019-06-20 at 09:54 -0700, groznykh wrote:

Hi all

I use GeoServer 2.15 and MongoDB 3.6

I've created a collection with data from  here
<[https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html](https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html)>  
(just two records), created 2dsphere index 

Then I created a layer from that collection and tried to request it by WMS
with cql_filter in URL params.

1. when I use simple attributes (that are not multiple) in the filter all is
fine (points are visible):

  - filter: "contract.mail" ILIKE '[%@mail.com](mailto:%@mail.com)%' OR "name" ILIKE '%station%'

  - mongo log:  { ... , filter: { }, projection: { geometry: 1,
contact.mail: 1, name: 1 }, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png)> 

2. when I use attributes that are multiple in the filter then it DOES NOT
work (there is no any point on received PNG):

  - filter: "measurements.name" ILIKE '%wind%' OR "measurements.unit" ILIKE
'%c%'

  - mongo log: { ... , filter: { }, projection: {geometry: 1,
measurements.unit: 1, measurements.name: 1}, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%281%29.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%281%29.png)> 

3. when I change OR to AND in previous filter then it's works fine:

  - filter: "measurements.name" ILIKE '%wind%' AND "measurements.unit" ILIKE
'%c%'

  - mongo log: { ..., filter: { $and: [ { measurements.name: /^.*wind.*$/i
}, { measurements.unit: /^.*p.*$/i } ] }, projection: { geometry: 1 }, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%282%29.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%282%29.png)> 

So I concluded that GeoServer does AND operation by sending filter to
mongoDB. And OR operation by itself requesting all records from collection.
But with attributes that are multiple something goes wrong. I think it's a
bug.

If you have any sugestions please tell me.

--
Sent from: [http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html](http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html)

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

Fernando_Mino · July 3, 2019, 1:13pm

Hi,

In fact the $or native operator is disabled on Geotools MongoDB module. This is because MongoDB until 2.4.9 version don’t support the mix of $or and spatial indexes very well.

Checking some MongoDB release notes seems like this was improved on 2.6 version:

$or and GeoSpatial Queries

Changed in version 2.6
$or supports geospatial clauses with the following exception for the near clause (near clause includes $nearSphere and $near). $or cannot contain a near clause with any other clause.

So if Geotools MongoDB module starts to support native $or clause by default this could lead to some bugs on previous versions. A good improvement could be a configuration for activating/deactivating native $or.

Regards,

Fernando Mino

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

==

Fernando Mino

Software Engineer

GeoSolutions S.A.S.

Via di Montramito 3/A

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

		From “groznykh” groznykh.dev@anonymised.com To geoserver-devel@lists.sourceforge.net Cc Date Thu, 20 Jun 2019 09:54:30 -0700 (MST) Subject [Geoserver-devel] CQL filter OR operator does not work on attributes that are multiple on MongoDB DataSource Hi all I use GeoServer 2.15 and MongoDB 3.6 I’ve created a collection with data from here https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html (just two records), created 2dsphere index Then I created a layer from that collection and tried to request it by WMS with cql_filter in URL params. 1. when I use simple attributes (that are not multiple) in the filter all is fine (points are visible): - filter: “contract.mail” ILIKE ‘%@mail.com%’ OR “name” ILIKE ‘%station%’ - mongo log: { … , filter: { }, projection: { geometry: 1, contact.mail: 1, name: 1 }, … } http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png 2. when I use attributes that are multiple in the filter then it DOES NOT work (there is no any point on received PNG): - filter: “measurements.name” ILIKE ‘%wind%’ OR “measurements.unit” ILIKE ‘%c%’ - mongo log: { … , filter: { }, projection: {geometry: 1, measurements.unit: 1, measurements.name: 1}, … } http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(1).png 3. when I change OR to AND in previous filter then it’s works fine: - filter: “measurements.name” ILIKE ‘%wind%’ AND “measurements.unit” ILIKE ‘%c%’ - mongo log: { …, filter: { $and: [ { measurements.name: /^.wind.$/i }, { measurements.unit: /^.p.$/i } ] }, projection: { geometry: 1 }, … } http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(2).png So I concluded that GeoServer does AND operation by sending filter to mongoDB. And OR operation by itself requesting all records from collection. But with attributes that are multiple something goes wrong. I think it’s a bug. If you have any sugestions please tell me. – Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html _______________________________________________ Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Nuno_Oliveira1 · July 9, 2019, 3:53pm

Hi Fernando,
thank you for the feedback. Linking GeoTools DEV mailing list since this actually turns out to be related whit GeoTools MongoDB store core.

So if I understood correctly, $or clause is correctly supported since MongoDB 2.6? which was released 5 years ago [1]?

GeoSolutions is using MogoDB driver:

<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>3.3.0</version>
    <type>jar</type>
    <optional>false</optional>
</dependency>

… 3.3.0 is only compatible with MongoDB 2.6 and more recent versions [2].

So +1 one for your suggestion Fernando:

activate $or support in GeoTools MongoDB store
add a Java property environment property that would allow the deactivation of the $or operator support, to be on the safe side

Best regards,

[1] https://www.mongodb.com/support-policy
[2] https://docs.mongodb.com/ecosystem/drivers/driver-compatibility-reference/

On Wed, 2019-07-03 at 15:13 +0200, fernando.mino@anonymised.com wrote:

Hi,

In fact the $or native operator is disabled on Geotools MongoDB module. This is because MongoDB until 2.4.9 version don’t support the mix of $or and spatial indexes very well.

Checking some MongoDB release notes seems like this was improved on 2.6 version:

$or and GeoSpatial Queries

Changed in version 2.6
$or supports geospatial clauses with the following exception for the near clause (near clause includes $nearSphere and $near). $or cannot contain a near clause with any other clause.

So if Geotools MongoDB module starts to support native $or clause by default this could lead to some bugs on previous versions. A good improvement could be a configuration for activating/deactivating native $or.

Regards,

Fernando Mino

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

==

Fernando Mino

Software Engineer

GeoSolutions S.A.S.

Via di Montramito 3/A

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

From “groznykh” groznykh.dev@anonymised.com
To geoserver-devel@lists.sourceforge.net
Cc
Date Thu, 20 Jun 2019 09:54:30 -0700 (MST)
Subject [Geoserver-devel] CQL filter OR operator does not work on attributes that are multiple on MongoDB DataSource

Hi all

I use GeoServer 2.15 and MongoDB 3.6

I’ve created a collection with data from here
https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html
(just two records), created 2dsphere index

Then I created a layer from that collection and tried to request it by WMS
with cql_filter in URL params.

1. when I use simple attributes (that are not multiple) in the filter all is
fine (points are visible):

- filter: “contract.mail” ILIKE ‘%@mail.com%’ OR “name” ILIKE ‘%station%’

- mongo log: { … , filter: { }, projection: { geometry: 1,
contact.mail: 1, name: 1 }, … }

http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png

2. when I use attributes that are multiple in the filter then it DOES NOT
work (there is no any point on received PNG):

- filter: “measurements.name” ILIKE ‘%wind%’ OR “measurements.unit” ILIKE
‘%c%’

- mongo log: { … , filter: { }, projection: {geometry: 1,
measurements.unit: 1, measurements.name: 1}, … }

http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(1).png

3. when I change OR to AND in previous filter then it’s works fine:

- filter: “measurements.name” ILIKE ‘%wind%’ AND “measurements.unit” ILIKE
‘%c%’

- mongo log: { …, filter: { $and: [ { measurements.name: /^.wind.$/i
}, { measurements.unit: /^.p.$/i } ] }, projection: { geometry: 1 }, … }

http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_(2).png

So I concluded that GeoServer does AND operation by sending filter to
mongoDB. And OR operation by itself requesting all records from collection.
But with attributes that are multiple something goes wrong. I think it’s a
bug.

If you have any sugestions please tell me.

–
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@anonymised.comsts.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

Artem_Groznykh · July 23, 2019, 7:57pm

Hi. As you offered I created a Jira ticket for this thread

	Без вирусов. www.avast.ru

пт, 21 июн. 2019 г. в 11:51, Nuno Oliveira <nuno.oliveira@anonymised.com8…>:

Hi,
thank you for the detailed feedback, indeed at first look this looks like a bug to me.

Can you create Jira ticket for this? Instructions:
http://geoserver.org/issues/

Best regards,
Nuno Oliveira

On Thu, 2019-06-20 at 09:54 -0700, groznykh wrote:

Hi all

I use GeoServer 2.15 and MongoDB 3.6

I've created a collection with data from  here
<[https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html](https://docs.geoserver.org/stable/en/user/data/app-schema/mongo-tutorial.html)>  
(just two records), created 2dsphere index 

Then I created a layer from that collection and tried to request it by WMS
with cql_filter in URL params.

1. when I use simple attributes (that are not multiple) in the filter all is
fine (points are visible):

  - filter: "contract.mail" ILIKE '[%@mail.com](mailto:%25@anonymised.com621...)%' OR "name" ILIKE '%station%'

  - mongo log:  { ... , filter: { }, projection: { geometry: 1,
contact.mail: 1, name: 1 }, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations.png)> 

2. when I use attributes that are multiple in the filter then it DOES NOT
work (there is no any point on received PNG):

  - filter: "[measurements.name](http://measurements.name)" ILIKE '%wind%' OR "measurements.unit" ILIKE
'%c%'

  - mongo log: { ... , filter: { }, projection: {geometry: 1,
measurements.unit: 1, [measurements.name](http://measurements.name): 1}, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%281%29.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%281%29.png)> 

3. when I change OR to AND in previous filter then it's works fine:

  - filter: "[measurements.name](http://measurements.name)" ILIKE '%wind%' AND "measurements.unit" ILIKE
'%c%'

  - mongo log: { ..., filter: { $and: [ { [measurements.name](http://measurements.name): /^.*wind.*$/i
}, { measurements.unit: /^.*p.*$/i } ] }, projection: { geometry: 1 }, ... }

<[http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%282%29.png](http://osgeo-org.1560.x6.nabble.com/file/t385081/stations-stations_%282%29.png)> 

So I concluded that GeoServer does AND operation by sending filter to
mongoDB. And OR operation by itself requesting all records from collection.
But with attributes that are multiple something goes wrong. I think it's a
bug.

If you have any sugestions please tell me.

--
Sent from: [http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html](http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html)

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit [http://goo.gl/it488V](http://goo.gl/it488V) for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

[http://www.geo-solutions.it](http://www.geo-solutions.it)
[http://twitter.com/geosolutions_it](http://twitter.com/geosolutions_it)

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

	Без вирусов. www.avast.ru

jive · July 26, 2019, 12:41am

Thanks for that - I migrated it to the correct project here GEOT-6348

I also created a MongoDB component in Jira so we could short list MongoDB issues.

···

–
Jody Garnett