[Geoserver-devel] CQL filter not working on MongoDB DataStore

Hi all,
I've been investigating an issue a customer of ours is experiencing with
the brand-new MongoDB community plugin (many thanks to Boundless :-)).

I'm using MongoDB version 3.0.3 and GeoServer/Geotools master. I've
created a MongoDB datastore and loaded the test data from Mongo's
2dsphere index tutorial:
http://docs.mongodb.org/manual/tutorial/build-a-2dsphere-index/

Basically, the points are displayed nicely in GeoServer's layer preview
page, but as soon as I add a CQL filter as simple as "name like ‘Central
%’", the features disappear (there should be one matching the filter).

I looked at the code and I think the problem is in method
toCursor(Query, List<Filter>) of MongoFeatureSource class. This method
builds the actual query that is sent to the database and returns a
DBCursor object to the caller. In doing so, it puts in a map the names
of all the attributes that should be returned by the query (the
equivalent of SELECT clause in SQL), but wrongly considers only the
attributes returned by Query.getPropertyNames(), ignoring the filters.
As a result, if a filter operates on a property that was not listed in
the query, it will inevitably fail, because that property will be
missing in the records returned by the cursor.

To fix the issue, I think it would be sufficient to include all the
properties mentioned in a filter in the list of the attributes returned
by the query.

Code for the fix is here:
https://github.com/geotools/geotools/compare/master...ridethepenguin:mongodb-cqlfilter-issue

Any thoughts?

--

Best regards,
Stefano Costa

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Dott. Stefano Costa
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

The approach seems sound, can you arrange a test case using the examples provided with the module?

···

On 12 June 2015 at 12:55, Stefano Costa <stefano.costa@anonymised.com> wrote:

Hi all,
I’ve been investigating an issue a customer of ours is experiencing with
the brand-new MongoDB community plugin (many thanks to Boundless :-)).

I’m using MongoDB version 3.0.3 and GeoServer/Geotools master. I’ve
created a MongoDB datastore and loaded the test data from Mongo’s
2dsphere index tutorial:
http://docs.mongodb.org/manual/tutorial/build-a-2dsphere-index/

Basically, the points are displayed nicely in GeoServer’s layer preview
page, but as soon as I add a CQL filter as simple as “name like ‘Central
%’”, the features disappear (there should be one matching the filter).

I looked at the code and I think the problem is in method
toCursor(Query, List) of MongoFeatureSource class. This method
builds the actual query that is sent to the database and returns a
DBCursor object to the caller. In doing so, it puts in a map the names
of all the attributes that should be returned by the query (the
equivalent of SELECT clause in SQL), but wrongly considers only the
attributes returned by Query.getPropertyNames(), ignoring the filters.
As a result, if a filter operates on a property that was not listed in
the query, it will inevitably fail, because that property will be
missing in the records returned by the cursor.

To fix the issue, I think it would be sufficient to include all the
properties mentioned in a filter in the list of the attributes returned
by the query.

Code for the fix is here:
https://github.com/geotools/geotools/compare/master…ridethepenguin:mongodb-cqlfilter-issue

Any thoughts?

Best regards,
Stefano Costa

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Dott. Stefano Costa
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.



Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel


Jody Garnett

Hi Jody,
here's the PR, complete with unit tests:
https://github.com/geotools/geotools/pull/879

On second thought, I realized that the issue I described previously
stems from the fact that PropertyIsLike was not listed among the filters
supported by MongoDataStore (which, I believe, was wrong, since there
was already support for translating it to Mongo in the FilterToMongo
class). As soon as I added PropertyIsLike to the datastore's
FilterCapabilities (and fixed a couple of issues in FilterToMongo), the
issue went away. Still, I believe the code I added to
MongoFeatureSource.toCursor() is substantially correct and so I left it
there.

I would love it if Tom, as the original author of the module, could
review my code before the PR is merged :slight_smile:

Also, I will need to backport this fix to GeoServer 2.7 (or 2.6, I must
check with the customer).

Il giorno ven, 12/06/2015 alle 13.01 -0700, Jody Garnett ha scritto:

The approach seems sound, can you arrange a test case using the
examples provided with the module?

--
Jody Garnett

On 12 June 2015 at 12:55, Stefano Costa
<stefano.costa@anonymised.com> wrote:
        Hi all,
        I've been investigating an issue a customer of ours is
        experiencing with
        the brand-new MongoDB community plugin (many thanks to
        Boundless :-)).
        
        I'm using MongoDB version 3.0.3 and GeoServer/Geotools master.
        I've
        created a MongoDB datastore and loaded the test data from
        Mongo's
        2dsphere index tutorial:
        http://docs.mongodb.org/manual/tutorial/build-a-2dsphere-index/
        
        Basically, the points are displayed nicely in GeoServer's
        layer preview
        page, but as soon as I add a CQL filter as simple as "name
        like ‘Central
        %’", the features disappear (there should be one matching the
        filter).
        
        I looked at the code and I think the problem is in method
        toCursor(Query, List<Filter>) of MongoFeatureSource class.
        This method
        builds the actual query that is sent to the database and
        returns a
        DBCursor object to the caller. In doing so, it puts in a map
        the names
        of all the attributes that should be returned by the query
        (the
        equivalent of SELECT clause in SQL), but wrongly considers
        only the
        attributes returned by Query.getPropertyNames(), ignoring the
        filters.
        As a result, if a filter operates on a property that was not
        listed in
        the query, it will inevitably fail, because that property will
        be
        missing in the records returned by the cursor.
        
        To fix the issue, I think it would be sufficient to include
        all the
        properties mentioned in a filter in the list of the attributes
        returned
        by the query.
        
        Code for the fix is here:
        https://github.com/geotools/geotools/compare/master...ridethepenguin:mongodb-cqlfilter-issue
        
        Any thoughts?

--

Best regards,
Stefano Costa

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Dott. Stefano Costa
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate.
Il loro utilizzo è consentito esclusivamente al destinatario del
messaggio, per le finalità indicate nel messaggio stesso. Qualora
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
cortesemente di darcene notizia via e-mail e di procedere alla
distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
Conservare il messaggio stesso, divulgarlo anche in parte,
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
diverse, costituisce comportamento contrario ai principi dettati dal
D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely
for the attention and use of the named addressee(s) and may be
confidential or proprietary in nature or covered by the provisions of
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
Data Protection Code).Any use not in accord with its purpose, any
disclosure, reproduction, copying, distribution, or either
dissemination, either whole or partial, is strictly forbidden except
previous formal approval of the named addressee(s). If you are not the
intended recipient, please contact immediately the sender by
telephone, fax or e-mail and delete the information in this message
that has been received in error. The sender does not give any warranty
or accept liability as the content, accuracy or completeness of sent
messages and accepts no responsibility for changes made after they
were sent or for other risks which arise as a result of e-mail
transmission, viruses, etc.

Hi guys,

I have a problem quite similar. I have this rule on the CQL filter
'myproperty > 0' and when I get the preview of my layer no results are
displayed.
The definition of the property is

/{ "localName" : "myproperty", "minOccurs" : 0, "maxOccurs" : 1, "type" :
            { "binding" : "java.lang.Double" },
       "userData" : { "mapping" : "myproperty" }
}/

and the corrispondent mongodb's query is:
/{ *"myproperty": { $gt: "0" }*, geometry: { $geoIntersects: { $geometry: {
type: "Polygon", coordinates: [ [ [ -36.92910748026556, 11.76353617582845 ],
[ -36.92910748026556, 71.55677632417155 ], [ 99.85879498026557,
71.55677632417155 ], [ 99.85879498026557, 11.76353617582845 ], [
-36.92910748026556, 11.76353617582845 ] ] ] } } } } /

Why is the zero treated as a String?

Thanks in advance
Best regards
Sante Barbuto

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/CQL-filter-not-working-on-MongoDB-DataStore-tp5210706p5307747.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

This appears to be a bug in the FilterToMongo class, where numbers are converted to Strings.

Feel free to add a bug report to the GeoTools JIRA, and/or create a pull request with a fix.

Torben

···

On Mon, Feb 13, 2017 at 8:38 AM, bard83 <bard83@anonymised.com> wrote:

Hi guys,

I have a problem quite similar. I have this rule on the CQL filter
‘myproperty > 0’ and when I get the preview of my layer no results are
displayed.
The definition of the property is

/{ “localName” : “myproperty”, “minOccurs” : 0, “maxOccurs” : 1, “type” :
{ “binding” : “java.lang.Double” },
“userData” : { “mapping” : “myproperty” }
}/

and the corrispondent mongodb’s query is:
/{ “myproperty”: { $gt: “0” }, geometry: { $geoIntersects: { $geometry: {
type: “Polygon”, coordinates: [ [ [ -36.92910748026556, 11.76353617582845 ],
[ -36.92910748026556, 71.55677632417155 ], [ 99.85879498026557,
71.55677632417155 ], [ 99.85879498026557, 11.76353617582845 ], [
-36.92910748026556, 11.76353617582845 ] ] ] } } } } /

Why is the zero treated as a String?

Thanks in advance
Best regards
Sante Barbuto


View this message in context: http://osgeo-org.1560.x6.nabble.com/CQL-filter-not-working-on-MongoDB-DataStore-tp5210706p5307747.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.


Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel