[Geoserver-users] Our experience with GeoServer and Solr as datastore

Hello all,
in our company we use with success GeoServer and Apache Solr for several project, but never togheter, so we tried to realize a proof of concept to demonstrate that solr can be a good choice as data for lat/lon positions managed by one of our projects (we are talking of the order of magnitude of about two MILLION or more positions stored).

We followed the guide that we found at: http://docs.geoserver.org/latest/en/user/community/solr/index.html

so we downloaded version 2.7-SNAPSHOT of GeoServer due to the fact that the guide is intended for 2.7.x versions.

We’re using Solr version 4.10.3 (> than 4.8/4.9 written on the guide).

We downloaded and successfully installed solr extension from http://ares.boundlessgeo.com/geoserver/master/community-latest/ (we was able to see Solr as possible datasource in GS admin page).

Until now everithing seems to be ok, but we had to work hard to see that “beautiful” little red squares representing our positions on the world rectangle.

And here comes first problems:
adding the “layer” or “categorization” field to ALL our solr documents to satisfy the requirement of the plugin was not easy, but fortunately we was “playing” with solr, so we was able to delete all created documents a recreate them with an additional default field, in our case named “layer” which default value is “positions”, so when uploading new documents, if a value for field “layer” is not provided the default value “positions” is set. In Solr you can do this by defining a field in this way in schema.xml file:

In our case was not a problem redefine the schema, but would be great if the solr plugin of geoserver uses the “layer” field as optional filter field and not as mandatory feature the user documents must provide.

Always in schema.xml we have a field:

where location_rpt is a type defined as:

Remember that this is not the default “location_rpt” field that comes with solr in bundle, we added the spatialContextFactory attribute, but to avoid classNotFound exceptions on Solr startup you have to install Jts libraries inside your solr web-inf/lib directory inside the war.

Following various guide on the internet we found that rpt fields can be write on solr product in different format: we used “longitude latitude” (longitude first and latitude after separated by a space) format. (THIS IS WRONG, read below for correct solution)

All done Solr side… for now…

And here comes the pain… in solr field configuration whe defined locationRpt as a “Geometry” (but also with ALL other possible choices in drop-down menu) but nothing: various exceptions both on solr output console (while geoserver does spatial queries on it) and on geoserver console (wrong geometry errors).

The SOLUTION was to define positionRpt content in WKT syntax, so a document with lat and lon values must have the rpt field expressed as “POINT(latitude,longitude)”, this is the syntax to express a point in WKT language.

And voilà! everithing worked, we was able to see ALL our positions on the layer preview.

Hoping to be helpful
Bye

Francesco Fornari
Italian Software Engineer

Hello, Francesco.

Thanks for your feedback. My organisation sponsored Geosolutions (Italy) to create the SOLR module, so I will discuss with Andrea Aime.

The reason for the ‘layer_name_field’ in the data source definition is to allow for many layers with one SOLR store, but if I understand correctly, you want to treat your whole SOLR store as ONE layer of points - is that correct?

Apologies for not being clear in the documentation that the spatial field needs to be in WKT format. We can rectify this. By the way, the spatial data can also be a line, polygon or multi-polygon - stored as a WKT string - although with my very brief experience with SOLR, I suggest that you would try to keep the number of vertices small.

Regards,

David Collins

Senior Geoscientist

Geological Survey of NSW, Australia.

···

On Wed, Jan 21, 2015 at 8:00 PM, Francesco Fornari <fra.fornari@anonymised.com> wrote:

Hello all,
in our company we use with success GeoServer and Apache Solr for several project, but never togheter, so we tried to realize a proof of concept to demonstrate that solr can be a good choice as data for lat/lon positions managed by one of our projects (we are talking of the order of magnitude of about two MILLION or more positions stored).

We followed the guide that we found at: http://docs.geoserver.org/latest/en/user/community/solr/index.html

so we downloaded version 2.7-SNAPSHOT of GeoServer due to the fact that the guide is intended for 2.7.x versions.

We’re using Solr version 4.10.3 (> than 4.8/4.9 written on the guide).

We downloaded and successfully installed solr extension from http://ares.boundlessgeo.com/geoserver/master/community-latest/ (we was able to see Solr as possible datasource in GS admin page).

Until now everithing seems to be ok, but we had to work hard to see that “beautiful” little red squares representing our positions on the world rectangle.

And here comes first problems:
adding the “layer” or “categorization” field to ALL our solr documents to satisfy the requirement of the plugin was not easy, but fortunately we was “playing” with solr, so we was able to delete all created documents a recreate them with an additional default field, in our case named “layer” which default value is “positions”, so when uploading new documents, if a value for field “layer” is not provided the default value “positions” is set. In Solr you can do this by defining a field in this way in schema.xml file:

In our case was not a problem redefine the schema, but would be great if the solr plugin of geoserver uses the “layer” field as optional filter field and not as mandatory feature the user documents must provide.

Always in schema.xml we have a field:

where location_rpt is a type defined as:

Remember that this is not the default “location_rpt” field that comes with solr in bundle, we added the spatialContextFactory attribute, but to avoid classNotFound exceptions on Solr startup you have to install Jts libraries inside your solr web-inf/lib directory inside the war.

Following various guide on the internet we found that rpt fields can be write on solr product in different format: we used “longitude latitude” (longitude first and latitude after separated by a space) format. (THIS IS WRONG, read below for correct solution)

All done Solr side… for now…

And here comes the pain… in solr field configuration whe defined locationRpt as a “Geometry” (but also with ALL other possible choices in drop-down menu) but nothing: various exceptions both on solr output console (while geoserver does spatial queries on it) and on geoserver console (wrong geometry errors).

The SOLUTION was to define positionRpt content in WKT syntax, so a document with lat and lon values must have the rpt field expressed as “POINT(latitude,longitude)”, this is the syntax to express a point in WKT language.

And voilà! everithing worked, we was able to see ALL our positions on the layer preview.

Hoping to be helpful
Bye

Francesco Fornari
Italian Software Engineer


New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet


Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Francesco it would be great to post your story to the geoserver blog (and any fun pictures you have of the result).

Jody

···

On 21 January 2015 at 01:00, Francesco Fornari <fra.fornari@anonymised.com> wrote:

Hello all,
in our company we use with success GeoServer and Apache Solr for several project, but never togheter, so we tried to realize a proof of concept to demonstrate that solr can be a good choice as data for lat/lon positions managed by one of our projects (we are talking of the order of magnitude of about two MILLION or more positions stored).

We followed the guide that we found at: http://docs.geoserver.org/latest/en/user/community/solr/index.html

so we downloaded version 2.7-SNAPSHOT of GeoServer due to the fact that the guide is intended for 2.7.x versions.

We’re using Solr version 4.10.3 (> than 4.8/4.9 written on the guide).

We downloaded and successfully installed solr extension from http://ares.boundlessgeo.com/geoserver/master/community-latest/ (we was able to see Solr as possible datasource in GS admin page).

Until now everithing seems to be ok, but we had to work hard to see that “beautiful” little red squares representing our positions on the world rectangle.

And here comes first problems:
adding the “layer” or “categorization” field to ALL our solr documents to satisfy the requirement of the plugin was not easy, but fortunately we was “playing” with solr, so we was able to delete all created documents a recreate them with an additional default field, in our case named “layer” which default value is “positions”, so when uploading new documents, if a value for field “layer” is not provided the default value “positions” is set. In Solr you can do this by defining a field in this way in schema.xml file:

In our case was not a problem redefine the schema, but would be great if the solr plugin of geoserver uses the “layer” field as optional filter field and not as mandatory feature the user documents must provide.

Always in schema.xml we have a field:

where location_rpt is a type defined as:

Remember that this is not the default “location_rpt” field that comes with solr in bundle, we added the spatialContextFactory attribute, but to avoid classNotFound exceptions on Solr startup you have to install Jts libraries inside your solr web-inf/lib directory inside the war.

Following various guide on the internet we found that rpt fields can be write on solr product in different format: we used “longitude latitude” (longitude first and latitude after separated by a space) format. (THIS IS WRONG, read below for correct solution)

All done Solr side… for now…

And here comes the pain… in solr field configuration whe defined locationRpt as a “Geometry” (but also with ALL other possible choices in drop-down menu) but nothing: various exceptions both on solr output console (while geoserver does spatial queries on it) and on geoserver console (wrong geometry errors).

The SOLUTION was to define positionRpt content in WKT syntax, so a document with lat and lon values must have the rpt field expressed as “POINT(latitude,longitude)”, this is the syntax to express a point in WKT language.

And voilà! everithing worked, we was able to see ALL our positions on the layer preview.

Hoping to be helpful
Bye

Francesco Fornari
Italian Software Engineer


New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet


Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users


Jody Garnett

On Wed, Jan 21, 2015 at 10:38 PM, David Collins <david.8.collins@anonymised.com>
wrote:

Hello, Francesco.

Thanks for your feedback. My organisation sponsored Geosolutions (Italy)
to create the SOLR module, so I will discuss with Andrea Aime.

The reason for the 'layer_name_field' in the data source definition is to
allow for many layers with one SOLR store, but if I understand correctly,
you want to treat your whole SOLR store as ONE layer of points - is that
correct?

Towards the end of that development I started to question myself if it
hadn't been better to allow the admin to stick a name, and provide a
generic filter
to identify the documents in that layer, similarly to how sql views are
created, but that would have required a significant redesign of both gui
and underlying code.
On the bright side, since the store is still in unsupported land, we'd
still be free to perform that change

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------