[Geoserver-devel] Any appetite for an ArcGIS ReST API datastore?

GeoServer GeoServer Developer
1

Folks,

We keep encountering public agencies that "open" their data by publishing them using the ArcGIS ReST API (on ArcGIS Online).

Therefore, we are toying with idea of developing a datastore to offer those data through WFS: has anyone ever considered this? (If so, any remarks/caveats?)

Cheers,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
LinkedIn: https://www.linkedin.com/in/lmorandini

2

Hi Luca,
while I haven’t worked on it, I’d certainly be happy to see such a datastore :slight_smile:

Cheers
Andrea

···

On Thu, Oct 27, 2016 at 8:14 AM, Luca Morandini <lmorandini@anonymised.com> wrote:

Folks,

We keep encountering public agencies that “open” their data by publishing them
using the ArcGIS ReST API (on ArcGIS Online).

Therefore, we are toying with idea of developing a datastore to offer those data
through WFS: has anyone ever considered this? (If so, any remarks/caveats?)

Cheers,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
LinkedIn: https://www.linkedin.com/in/lmorandini

The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

Sounds great, I have considered what is needed. A WMS cascade, or access features for WFS, WPS.

···

On Thu, Oct 27, 2016 at 8:14 AM, Luca Morandini <lmorandini@anonymised.com> wrote:

Folks,

We keep encountering public agencies that “open” their data by publishing them
using the ArcGIS ReST API (on ArcGIS Online).

Therefore, we are toying with idea of developing a datastore to offer those data
through WFS: has anyone ever considered this? (If so, any remarks/caveats?)

Cheers,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
LinkedIn: https://www.linkedin.com/in/lmorandini

The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

On 28/10/16 08:26, Jody Garnett wrote:

Sounds great, I have considered what is needed. A WMS cascade, or access features
for WFS, WPS.

The ArcGIS ReST API is a complex beast that includes the functionality of (almost) the whole range of OGC Servicesl. However, we are interested only in implementing a subset of them (Catalog Service and Feature Service).

An added complication is the proprietary JSON format ESRI uses for geometry, which is not GeoJSON. (ESRI kindly put out a JavaScript library to convert between the two formats, which should ease the pain.)

ESRI implemented as well client libraries in Python and Node.js that can be of some use.

Anyway, I gather no one has developed something suitable for a GeoServer datastore, but there is some interest in developing one.

Cheers,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
LinkedIn: https://www.linkedin.com/in/lmorandini

5

Yes there is interest - it would offer a “bridge” to organizations that are shutting down ArcSDE (and enable them to continue to use GeoServer). Frankly it may scale better than ArcSDE…

Note the work would actually take place over on the geotools-devel list (used for data i/o etc… and clients!).

···

On 30 October 2016 at 21:57, Luca Morandini <lmorandini@anonymised.com> wrote:

On 28/10/16 08:26, Jody Garnett wrote:

Sounds great, I have considered what is needed. A WMS cascade, or access features
for WFS, WPS.

The ArcGIS ReST API is a complex beast that includes the functionality of (almost)
the whole range of OGC Servicesl. However, we are interested only in implementing
a subset of them (Catalog Service and Feature Service).

An added complication is the proprietary JSON format ESRI uses for geometry, which
is not GeoJSON. (ESRI kindly put out a JavaScript library to convert between the
two formats, which should ease the pain.)

ESRI implemented as well client libraries in Python and Node.js that can be of
some use.

Anyway, I gather no one has developed something suitable for a GeoServer
datastore, but there is some interest in developing one.

Cheers,

Luca Morandini
Data Architect - AURIN project
Melbourne eResearch Group
Department of Computing and Information Systems
University of Melbourne
Tel. +61 03 903 58 380
Skype: lmorandini
LinkedIn: https://www.linkedin.com/in/lmorandini

The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

6

On Mon, Oct 31, 2016 at 7:14 AM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Yes there is interest - it would offer a "bridge" to organizations that
are shutting down ArcSDE (and enable them to continue to use GeoServer).
Frankly it may scale better than ArcSDE...

This is interesting... care to elaborate?

When we did the benchmarking sessions back at FOSS4G 2010 we noticed SDE
was not adding any measurable
overhead compared to direct access to the same spatial DB (at all tested
load levels), and we were actually
surprised.

I sort of doubt a JSON based transport protocol could be competitive with
that performance wise, but happy
to be proved wrong of course :slight_smile:

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

7

The arc sde client jar is not thread safe, so does not scale the best (even though we have done a great job only letting a single thread access at a connection at a time).

···

On Mon, Oct 31, 2016 at 7:14 AM, Jody Garnett <jody.garnett@anonymised.com> wrote:

Yes there is interest - it would offer a “bridge” to organizations that are shutting down ArcSDE (and enable them to continue to use GeoServer). Frankly it may scale better than ArcSDE…

This is interesting… care to elaborate?

When we did the benchmarking sessions back at FOSS4G 2010 we noticed SDE was not adding any measurable
overhead compared to direct access to the same spatial DB (at all tested load levels), and we were actually
surprised.

I sort of doubt a JSON based transport protocol could be competitive with that performance wise, but happy
to be proved wrong of course :slight_smile:

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

8

On Mon, Oct 31, 2016 at 3:48 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

The arc sde client jar is not thread safe, so does not scale the best
(even though we have done a great job only letting a single thread access
at a connection at a time).

I remember it differently... Gabriel implemented it so that multiple
readers could share the same connection
because of the high (monetary) price of each and the consequent normally
very small number of connections
available for GeoServer.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------