[Geoserver-devel] CSV data store as a stand alone module

GeoServer GeoServer Developer
1

Hi,
in GeoServer master branch, while adding the importer extension, we also get
as a side effect the CSV data store, which starts showing up in the “new datastore”
page.

There is one first gotcha, which is, the datastore was completely unusable until
https://github.com/geotools/geotools/pull/861 got merged (just did), but even with

that one, it’s still pretty hard to use since the datastore config panel is not
exactly providing useful hints on how to configure them.
It’s also undocumented.

I think it would be a good win to make a config panel for it, CSV sources are relatively
common anyways, and sometimes people just don’t want to load them into
a more efficient storage engine.

But then… where to put it? Its own extension module? Or directly in core maybe?
Telling people “ah yes, if you want the CSV datastore, just install the importer
extension” does not sound exactly great to me

Cheers
Andrea

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

On Tue, Jun 2, 2015 at 9:15 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

I think it would be a good win to make a config panel for it

As in, a custom one, since some params are making sense only for some
choice of mapping
strategy

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

One warning - Last time I interacted with the CSV datastore (in March) and GeoServer importer, it looked like it was still missing a few things. It seems the CSV datastore does not handle geometry columns especially well. CSV Import via the GeoServer importer UI only supports CSVs as lists of attributes (no geometry). Via the Importer REST API you can specify Lat/Lon columns. Support for WKT columns exists, but there is no way of actually importing/specifiying columns as WKT as far as I can see, although this might also be doable using the REST API.

A custom config panel seems like a very good idea, and should include some way to specify the Geometry column(s) (Lat/Lon or WKT).

Given that Property datastores are supported in core GeoServer, it would make just as much sense to also include the CSV datastore in core GeoServer.

Torben

···

On Tue, Jun 2, 2015 at 12:21 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Tue, Jun 2, 2015 at 9:15 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I think it would be a good win to make a config panel for it

As in, a custom one, since some params are making sense only for some choice of mapping
strategy

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

On Tue, Jun 2, 2015 at 6:13 PM, Torben Barsballe <
tbarsballe@anonymised.com> wrote:

One warning - Last time I interacted with the CSV datastore (in March) and
GeoServer importer, it looked like it was still missing a few things. It
seems the CSV datastore does not handle geometry columns especially well.
CSV Import via the GeoServer importer UI only supports CSVs as lists of
attributes (no geometry).

This is because the code makes no attempt to figure out which strategy to
use, it could just parse the file once to figure out the best strategy, and
then again to actually
import it. That's what I did for the GML reader, where there is no good way
to just read the headers and guess the GML version, sometimes the geometry
type, and its CRS (under the common case that the schema is nowhere to be
found)

Via the Importer REST API you can specify Lat/Lon columns. Support for WKT
columns exists, but there is no way of actually importing/specifiying
columns as WKT as far as I can see, although this might also be doable
using the REST API.

Hum... I don't know, never tried. The example I added in the docs indeed
uses a transform (post datastore read) to
make points out of lat/lon, even if the store should be perfectly happy to
read that directly as points (if we just told
it to use the right strategy).

A custom config panel seems like a very good idea, and should include some
way to specify the Geometry column(s) (Lat/Lon or WKT).

Yep, if you look at my CSV store patch, I exposed already the relevant
fields along with the namespace param

Given that Property datastores
<http://docs.geoserver.org/latest/en/user/data/app-schema/data-stores.html#property-file&gt;
are supported in core GeoServer, it would make just as much sense to also
include the CSV datastore in core GeoServer.

That's what I was thinking too

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

On Tue, Jun 2, 2015 at 10:15 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Tue, Jun 2, 2015 at 6:13 PM, Torben Barsballe <
tbarsballe@anonymised.com> wrote:

One warning - Last time I interacted with the CSV datastore (in March)
and GeoServer importer, it looked like it was still missing a few things.
It seems the CSV datastore does not handle geometry columns especially
well. CSV Import via the GeoServer importer UI only supports CSVs as lists
of attributes (no geometry).

This is because the code makes no attempt to figure out which strategy to
use, it could just parse the file once to figure out the best strategy, and
then again to actually
import it. That's what I did for the GML reader, where there is no good
way to just read the headers and guess the GML version, sometimes the
geometry type, and its CRS (under the common case that the schema is
nowhere to be found)

I have done a bit of work trying to get this working for CSV stores in the
GeoServer importer, although I haven't got anything final yet. I may get
this working sometime in the future.

Torben

6

CSVDataStore is used as a tutorial in GeoTools (as was PropertyDataStore before it).

I think it could use a QA/Readability review for the different CSV reading strategies, I found the code to be amazing convoluted to follow (with some API choices made so test cases could be written against strings).

···

On 2 June 2015 at 12:13, Torben Barsballe <tbarsballe@anonymised.com> wrote:

One warning - Last time I interacted with the CSV datastore (in March) and GeoServer importer, it looked like it was still missing a few things. It seems the CSV datastore does not handle geometry columns especially well. CSV Import via the GeoServer importer UI only supports CSVs as lists of attributes (no geometry). Via the Importer REST API you can specify Lat/Lon columns. Support for WKT columns exists, but there is no way of actually importing/specifiying columns as WKT as far as I can see, although this might also be doable using the REST API.

A custom config panel seems like a very good idea, and should include some way to specify the Geometry column(s) (Lat/Lon or WKT).

Given that Property datastores are supported in core GeoServer, it would make just as much sense to also include the CSV datastore in core GeoServer.

Torben

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

On Tue, Jun 2, 2015 at 12:21 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Tue, Jun 2, 2015 at 9:15 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I think it would be a good win to make a config panel for it

As in, a custom one, since some params are making sense only for some choice of mapping
strategy

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

7

On Wed, Jun 3, 2015 at 2:03 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

CSVDataStore is used as a tutorial in GeoTools (as was PropertyDataStore
before it).

I think it could use a QA/Readability review for the different CSV reading
strategies, I found the code to be amazing convoluted to follow (with some
API choices made so test cases could be written against strings).

If more QA is needed (and there are no resources for it), I suggest we find
a way to un-register it so that it does not show up in the GeoServer UI when
using the importer module, exposing a non fully functioning store is not a
desirable side effect of installing the importer.
Hum... I guess we'd need some sort of blacklist for stores?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

8

I guess we would, let me look at what is required for creating a data store page (not something I have done before). Even if we just expose the lat/long mapping it would be better / less confusing then what is now available.

···

On 3 June 2015 at 01:46, Andrea Aime <andrea.aime@anonymised.com> wrote:

–
Jody Garnett

On Wed, Jun 3, 2015 at 2:03 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

CSVDataStore is used as a tutorial in GeoTools (as was PropertyDataStore before it).

I think it could use a QA/Readability review for the different CSV reading strategies, I found the code to be amazing convoluted to follow (with some API choices made so test cases could be written against strings).

If more QA is needed (and there are no resources for it), I suggest we find a way to un-register it so that it does not show up in the GeoServer UI when
using the importer module, exposing a non fully functioning store is not a desirable side effect of installing the importer.
Hum… I guess we’d need some sort of blacklist for stores?

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.