[Geoserver-devel] Any documentation for the importer REST api?

GeoServer GeoServer Developer
1

Hi,
the importer module has a REST API, which apparently can do the same as the GUI, and more,
in a more automation friendly way.

However, there is no documentation, there is a few tests, but they look a bit programmatic
and the workflow they suggest seems sort of a multi-step one, something like:

  • create a import
  • populate its tasks
  • actually start the import
  • monitor its progress

But… well, I could be misunderstanding.

Soo… is there any published doc about the topic?
Or someone will have to reverse engineer what’s going on starting from the tests?

Cheers
Andrea

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Hi Andrea,

···

On Thu, Feb 12, 2015 at 12:31 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
the importer module has a REST API, which apparently can do the same as the GUI, and more,
in a more automation friendly way.

Correct. One major limitation of the wicket UI for the importer is that the data must be local.

However, there is no documentation, there is a few tests, but they look a bit programmatic

and the workflow they suggest seems sort of a multi-step one, something like:

  • create a import
  • populate its tasks
  • actually start the import
  • monitor its progress

Correct - it is a multi-step process. The idea is that a user might upload a variety of data in a single zip file. This results in multiple tasks that may or may not need configuration.

But… well, I could be misunderstanding.

Soo… is there any published doc about the topic?

There is some in the docs directory - it may be outdated, however.

Or someone will have to reverse engineer what’s going on starting from the tests?

I had done some work to create a java client to make descriptive test cases easier to write and execute but never completed it, but…

There is also a client library here and an integration test suite with a narrower focus: https://github.com/boundlessgeo/gsimporter

Hope that helps

–

Ian Schneider
Software Engineer | Boundless
ischneider@anonymised.com…3839…
1-877-673-6436

@boundlessgeo

3

On Thu, Feb 12, 2015 at 6:33 PM, Ian Schneider <ischneider@anonymised.com>
wrote:

But.. well, I could be misunderstanding.

Soo... is there any published doc about the topic?

There is some in the docs directory - it may be outdated, however.

Doc doc... oh!
https://github.com/geoserver/geoserver/tree/master/src/extension/importer/doc

Did not see this one :slight_smile:
Interesting, I'll see if/how I can merge that into our docs

Or someone will have to reverse engineer what's going on starting from
the tests?

I had done some work to create a java client to make descriptive test
cases easier to write and execute but never completed it, but...

There is also a client library here and an integration test suite with a
narrower focus: https://github.com/boundlessgeo/gsimporter

Thanks for the link

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

4

On Mon, Feb 16, 2015 at 6:47 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Doc doc... oh!

https://github.com/geoserver/geoserver/tree/master/src/extension/importer/doc

Did not see this one :slight_smile:
Interesting, I'll see if/how I can merge that into our docs

Ok, so a few questions if I may.

When creating a new import with proposed id with PUT, the code says:
*"Create import with proposed id <importId>. If the proposed id is ahead of
the current (next) id, the current id will be advanced. If the proposed id
is less than or equal to the current id, the current will be used. This
allows an external system to dictate the id management."*

Soo... basically the id will be accepted only if it is greater than the
current id, and the current will be updated to it, right?

Besides that, one thing standing out is that the rst version of the doc
says tasks has items, whilst the markdown
one says tasks has layers, directories, files, but no items (and also cites
the transforms, nice)

Looking at the rest mappings declared in the applicationContext.xml it
looks like items are not there, but layers
are, so the API reference table in the rst document needs to be updated

One thing that I was wondering is, is it not possible to just POST the
whole json/xml tree describing
the import, its tasks, it transforms and whatnot in a single shot? Say for
example that the upload
(maybe large) was managed out of band with a different tool (FTP,
resumable/chunked HTTP),
I see no reason for making all this little calls to stand up a complex
import.
Am I missing something?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Hi,
another one. It seems the representation of resources is stuck to json, or at least,
it doe not change when changing the suffix of the resource to .xml.

This is inconsistent to the GeoServer rest api, I think it should be reported as a bug
(also given the code is using the geoserver resource representation for some of its
api bits)

Cheers
Andrea

···

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

6

Hi Andrea,

···

On Mon, Feb 16, 2015 at 11:34 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

I only realized the other day when you asked that these don’t get included into the doc build.

Right.

I cannot recall how we ended up w/ two docs (Justin?) but the item concept is now removed.

I believe it’s possible but have not tried.

The API is complex, agreed. As for the little calls, performance wise, they are generally not noticeable. Usability wise, annoying, yes.

The idea was/is to support an interactive workflow:

user: here’s my data…
gs: ok, I see you’ve given me 3 shapefiles, one with a bad/missing projection and something I don’t recognize/support
user: ok, ignore that other thing and use 4326 for that bad shapefile
user: now that I see my ‘time’ attribute is actually encoded as text, convert that to a date
user: oh yeah, and import that 1st shapefile into the database but call it something else and create an index on that attribute
…

If you know what you want to do ahead of time, this should ideally be supported as well.

There was initially more ‘post’ import functionality present at one point (layer configuration, etc.) but this seems like it belongs in the standard catalog API.

Hope that explains some.

–

On Mon, Feb 16, 2015 at 6:47 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Doc doc… oh!
https://github.com/geoserver/geoserver/tree/master/src/extension/importer/doc

Did not see this one :slight_smile:
Interesting, I’ll see if/how I can merge that into our docs

Ok, so a few questions if I may.

When creating a new import with proposed id with PUT, the code says:
“Create import with proposed id . If the proposed id is ahead of the current (next) id, the current id will be advanced. If the proposed id is less than or equal to the current id, the current will be used. This allows an external system to dictate the id management.”

Soo… basically the id will be accepted only if it is greater than the current id, and the current will be updated to it, right?

Besides that, one thing standing out is that the rst version of the doc says tasks has items, whilst the markdown
one says tasks has layers, directories, files, but no items (and also cites the transforms, nice)

Looking at the rest mappings declared in the applicationContext.xml it looks like items are not there, but layers
are, so the API reference table in the rst document needs to be updated

One thing that I was wondering is, is it not possible to just POST the whole json/xml tree describing
the import, its tasks, it transforms and whatnot in a single shot? Say for example that the upload
(maybe large) was managed out of band with a different tool (FTP, resumable/chunked HTTP),
I see no reason for making all this little calls to stand up a complex import.

Am I missing something?

Ian Schneider
Software Engineer | Boundless
ischneider@anonymised.com
1-877-673-6436

@boundlessgeo