[Geoserver-users] Custom InfluxDB datastore - implementing filtering

Hi,

So I've managed to make DataStore that connects to InfuxDB and fetches the
data. Now, I would like to implement filtering with CQL, so that I don't
fetch all data from DB, but only the ones that I specify.
Specifically, based on the parameters I enter (for example 4/10 parameters),
I would execute specific InfluxQL query on database and fetch data with only
those 4 parameters.

I think that postgis plugin works that way, you specify the parameters in
CQL and from it the SQL query is created and only specified data is fetched.
I want to accomplish basically the same thing.

I was wondering is there any tutorial/documentation for this, where I can
find how to implement this kind of filtering? What is the best approach
regarding this?

Best regards,
Maja

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

You need to look at (for example) PostgisFilterToSQL - which is where the generic GeoTools filter is converted to the database specific SQL query. The *FilterToSQL classes are written as visitors which walk down the filter tree and carry out an operation at each node of the tree. Other DB specific parts of the query (encoding geometry etc) are handled in the Dialect class.

IFAIK there is no tutorial for this as it isn’t something that gets done very often, your best bet is to discuss any problems you run in to on the GeoTools lists.

Ian

···

On 17 January 2017 at 11:48, Firefly <maja.filakovic@anonymised.com> wrote:

Hi,

So I’ve managed to make DataStore that connects to InfuxDB and fetches the
data. Now, I would like to implement filtering with CQL, so that I don’t
fetch all data from DB, but only the ones that I specify.
Specifically, based on the parameters I enter (for example 4/10 parameters),
I would execute specific InfluxQL query on database and fetch data with only
those 4 parameters.

I think that postgis plugin works that way, you specify the parameters in
CQL and from it the SQL query is created and only specified data is fetched.
I want to accomplish basically the same thing.

I was wondering is there any tutorial/documentation for this, where I can
find how to implement this kind of filtering? What is the best approach
regarding this?

Best regards,
Maja


View this message in context: http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706.html
Sent from the GeoServer - User mailing list archive at Nabble.com.


Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


Geoserver-users mailing list
Geoserver-users@anonymised.com.382…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Ian Turton

Hi Maja,

Ian’s dead-on. If you are calling to InfluxDB directly, your options are to handle certain filters manually (which is great for a prototype!) or implement a visitor (again, you can focus on certain predicates first).

As an alternative solution, it appears that there is at least one JDBC connector for InfluxDB (https://github.com/influxdata/influxdb-java). You could plug that JDBC connector into the general GeoTools JDBCDataStore by mirroring what is done in the PostGIS DataStore.

The advantage of the first approach is that you’d be plumbing things directly, and it might be more clear what is happening. With the JDBC approach, the general GeoTools FilterToSQL code would do lots of the heavy lifting for you, so that’s win. On the downside, you’d have to learn an extra level of abstraction and you’d be reliant on the JDBC implementation.

As Ian said, this is a somewhat uncommon task, so asking on the lists would be a great way to get more help! Also, I hope you can share your implementation; wiring up InfluxDB to GeoServer sounds like a great project!

Cheers,

Jim

As a concrete example of a non-trivial, non-sql visitor, check out this class from ElasticGeo: https://github.com/ngageoint/elasticgeo/blob/master/gt-elasticsearch-parent/gt-elasticsearch-core/src/main/java/mil/nga/giat/data/elasticsearch/FilterToElastic.java.

If you are willing to read Scala, check out this which maps CQL to CQEngine queries: https://github.com/locationtech/geomesa/blob/master/geomesa-memory/geomesa-cqengine/src/main/scala/org/locationtech/geomesa/memory/cqengine/utils/CQEngineQueryVisitor.scala with unit tests: https://github.com/locationtech/geomesa/blob/master/geomesa-memory/geomesa-cqengine/src/test/scala/org/locationtech/geomesa/memory/cqengine/utils/CQEngineQueryVisitorTest.scala. GeoMesa has a few other visitors here and there; I think that one is the easiest to read.

If you haven’t seen it, this tutorial should help with the general datastore bits: http://docs.geotools.org/stable/userguide/tutorial/datastore/index.html

···

On 01/17/2017 08:23 AM, Ian Turton wrote:

You need to look at (for example) PostgisFilterToSQL - which is where the generic GeoTools filter is converted to the database specific SQL query. The *FilterToSQL classes are written as visitors which walk down the filter tree and carry out an operation at each node of the tree. Other DB specific parts of the query (encoding geometry etc) are handled in the Dialect class.

IFAIK there is no tutorial for this as it isn’t something that gets done very often, your best bet is to discuss any problems you run in to on the GeoTools lists.

Ian

On 17 January 2017 at 11:48, Firefly <maja.filakovic@anonymised.com> wrote:

Hi,

So I’ve managed to make DataStore that connects to InfuxDB and fetches the
data. Now, I would like to implement filtering with CQL, so that I don’t
fetch all data from DB, but only the ones that I specify.
Specifically, based on the parameters I enter (for example 4/10 parameters),
I would execute specific InfluxQL query on database and fetch data with only
those 4 parameters.

I think that postgis plugin works that way, you specify the parameters in
CQL and from it the SQL query is created and only specified data is fetched.
I want to accomplish basically the same thing.

I was wondering is there any tutorial/documentation for this, where I can
find how to implement this kind of filtering? What is the best approach
regarding this?

Best regards,
Maja


View this message in context: http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706.html
Sent from the GeoServer - User mailing list archive at Nabble.com.


Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


Geoserver-users mailing list
Geoserver-users@anonymised.comsourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Ian Turton

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! [http://sdm.link/slashdot](http://sdm.link/slashdot)
_______________________________________________
Geoserver-users mailing list
[Geoserver-users@lists.sourceforge.net](mailto:Geoserver-users@anonymised.comsourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-users](https://lists.sourceforge.net/lists/listinfo/geoserver-users)

Hi all,

thank you for both your inputs. I'll look into that and try to implement
this. Right now, implementation of this is on the hold, but I hope that I'll
finish someday in the near future, when I'll have more free time.

Thanks again.
Maja

Jim Hughes wrote

Hi Maja,

Ian's dead-on. If you are calling to InfluxDB directly, your options
are to handle certain filters manually (which is great for a prototype!)
or implement a visitor (again, you can focus on certain predicates first).

As an alternative solution, it appears that there is at least one JDBC
connector for InfluxDB (https://github.com/influxdata/influxdb-java).
You could plug that JDBC connector into the general GeoTools
JDBCDataStore by mirroring what is done in the PostGIS DataStore.

The advantage of the first approach is that you'd be plumbing things
directly, and it might be more clear what is happening. With the JDBC
approach, the general GeoTools FilterToSQL code would do lots of the
heavy lifting for you, so that's win. On the downside, you'd have to
learn an extra level of abstraction and you'd be reliant on the JDBC
implementation.

As Ian said, this is a somewhat uncommon task, so asking on the lists
would be a great way to get more help! Also, I hope you can share your
implementation; wiring up InfluxDB to GeoServer sounds like a great
project!

Cheers,

Jim

As a concrete example of a non-trivial, non-sql visitor, check out this
class from ElasticGeo:
https://github.com/ngageoint/elasticgeo/blob/master/gt-elasticsearch-parent/gt-elasticsearch-core/src/main/java/mil/nga/giat/data/elasticsearch/FilterToElastic.java.

If you are willing to read Scala, check out this which maps CQL to
CQEngine queries:
https://github.com/locationtech/geomesa/blob/master/geomesa-memory/geomesa-cqengine/src/main/scala/org/locationtech/geomesa/memory/cqengine/utils/CQEngineQueryVisitor.scala
with unit tests:
https://github.com/locationtech/geomesa/blob/master/geomesa-memory/geomesa-cqengine/src/test/scala/org/locationtech/geomesa/memory/cqengine/utils/CQEngineQueryVisitorTest.scala.
GeoMesa has a few other visitors here and there; I think that one is the
easiest to read.

If you haven't seen it, this tutorial should help with the general
datastore bits:
http://docs.geotools.org/stable/userguide/tutorial/datastore/index.html

On 01/17/2017 08:23 AM, Ian Turton wrote:

You need to look at (for example) PostgisFilterToSQL - which is where
the generic GeoTools filter is converted to the database specific SQL
query. The *FilterToSQL classes are written as visitors which walk
down the filter tree and carry out an operation at each node of the
tree. Other DB specific parts of the query (encoding geometry etc)
are handled in the Dialect class.

IFAIK there is no tutorial for this as it isn't something that gets
done very often, your best bet is to discuss any problems you run in
to on the GeoTools lists.

Ian

On 17 January 2017 at 11:48, Firefly &lt;

maja.filakovic@

&gt; &lt;mailto:

maja.filakovic@

&gt;> wrote:

    Hi,

    So I've managed to make DataStore that connects to InfuxDB and
    fetches the
    data. Now, I would like to implement filtering with CQL, so that I
    don't
    fetch all data from DB, but only the ones that I specify.
    Specifically, based on the parameters I enter (for example 4/10
    parameters),
    I would execute specific InfluxQL query on database and fetch data
    with only
    those 4 parameters.

    I think that postgis plugin works that way, you specify the
    parameters in
    CQL and from it the SQL query is created and only specified data
    is fetched.
    I want to accomplish basically the same thing.

    I was wondering is there any tutorial/documentation for this,
    where I can
    find how to implement this kind of filtering? What is the best
    approach
    regarding this?

    Best regards,
    Maja

    --
    View this message in context:
   
http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706.html
   
&lt;http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706.html&gt;
    Sent from the GeoServer - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
    Check out the vibrant tech community on one of the world's most
    engaging tech sites, SlashDot.org! http://sdm.link/slashdot
    _______________________________________________
    Geoserver-users mailing list
    
Geoserver-users@anonymised.com

    &lt;mailto:

Geoserver-users@anonymised.com

&gt;

    https://lists.sourceforge.net/lists/listinfo/geoserver-users
    &lt;https://lists.sourceforge.net/lists/listinfo/geoserver-users&gt;

--
Ian Turton

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-users mailing list

Geoserver-users@anonymised.com

https://lists.sourceforge.net/lists/listinfo/geoserver-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Geoserver-users mailing list

Geoserver-users@anonymised.com

https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Custom-InfluxDB-datastore-implementing-filtering-tp5303706p5304003.html
Sent from the GeoServer - User mailing list archive at Nabble.com.