[Geoserver-devel] [GEOS-9886] yaml schemas definitions aren't compatible with OpenAPI (Swagger 2.0) definitions. Generated Swagger clients don't work.

GeoServer GeoServer Developer
1

Hello,

/workspaces.yaml/ file on docs page cannot be used to generate a client by
the mean of /Swagger/ client generator.

to summarize : their examples (human written) tells "/create a workspace by
posting {workspace: {name: 'test'}}/" but /swagger-ui.html/ generated for
/workspaces.yaml/ and available too on doc page shows that this /yaml/ would
propose to send a /{name: 'test'}/ instead.

There's a mismatch, and if it is generated by Swagger from this /yaml/ file,
a /Geoserver/ REST client will causes an HTTP 500 error on server side.

I've corrected the /yaml/ declaration for the example of the
/postWorkspaces/ operation, but I don't know in what project I could put
that corrected yaml with a test to show the validity of the correction.
(This will be my first commit on geoserver (fork)).

workspaces.yaml
<http://osgeo-org.1560.x6.nabble.com/file/t365683/workspaces.yaml&gt;

Regards,

Marc Le Bihan

--
Sent from: http://osgeo-org.1560.x6.nabble.com/GeoServer-Dev-f3819232.html

2

I'm not following this 100% (not very familiar with this part of the cde), but if I got what you are looking for:

https://github.com/geoserver/geoserver/blob/master/doc/en/api/1.0.0/workspaces.yaml

I guess the test would go into WorkspaceTest:

https://github.com/geoserver/geoserver/blob/master/src/restconfig/src/test/java/org/geoserver/rest/catalog/WorkspaceTest.java

although if its very complex or needs unique setup, you could create a new Test in that area.

Brad

3

Thanks,

I red the WorkspaceTest.java and understand more the trouble. The test doesn't consider workspaces.yaml declarations to produce its client request when it's run.
I think no tool or no part of Geoserver use the content of the yaml rest config files at the moment.

This is why I wasn't able to find after Geoserver startup :
http://localhost:8080/geoserver/swagger-ui.html giving REST html documentation.
http://localhost:8080/geoserver/v3/api-docs giving Open Api json output for swagger/open api client generation.

and I was expecting finding them, because they might change between Geoserver application versions (for example : yaml will change from 1.8.2 to 1.9.x if my correction is applied.
If yaml are kept on blob/master/doc/en/api and maintained only there,
https://github.com/geoserver/geoserver/blob/master/doc/en/api/1.0.0/workspaces.yaml would lead to versions from 1.2.0 to 1.8.2 and https://github.com/geoserver/geoserver/blob/master/doc/en/api/1.0.1/workspaces.yaml to 1.9.0 and later ?
It's hard to stand).

For testing, I need a separate and new intergration test.
Do those run with a remoteOwsTests or a restconfig maven profile ?

I would then wrote a simple test :
1. Generate client classes from workspace.yaml file at test-compile time (when tests are ran under the maven profile required) :
2. Start intergration instance ot Geoserver : are they easy to start ?
3. Call it through the client test code.

Marc.

On 07/02/2021 00:04, Brad Hards wrote:

I'm not following this 100% (not very familiar with this part of the cde), but if I got what you are looking for:

https://github.com/geoserver/geoserver/blob/master/doc/en/api/1.0.0/workspaces.yaml

I guess the test would go into WorkspaceTest:

https://github.com/geoserver/geoserver/blob/master/src/restconfig/src/test/java/org/geoserver/rest/catalog/WorkspaceTest.java

although if its very complex or needs unique setup, you could create a new Test in that area.

Brad

4

Correct, they have been hand-written as a documentation effort during the 2017 code sprint.
Generating a client was not a goal back then, it’s still not today (they are not really being maintained all that much).

There is also a separate issue that would need to be handled, that is, the REST representation
changes as soon as one of the internal config objects (CatalogInfo implementers) changes, which
happens every one and then, but without anyone ever caring to update the YAML files.
The changes are always made so that new fields are not mandatory, in order to allow backwards
compatibility with the old stored XML files.
Nothing in REST needs to be touched, as the representations are build using
a reflection like mechanism (XStream in particular), but a client written in a type safe language might
have issues adapting.

As far as I know, core devs are not generating clients, and my colleagues that are using the REST
API do so using a hand-written python library, https://github.com/GeoNode/geoserver-restconfig
I guess these clients are tolerant to representation changing,

I believe having YAML files that can be used to generate clients from is a worthy goal, but
we need more developers to get involved in the project and take care of that part, either

by generating the yamls from code (to keep the aligned at all time), or trying to follow the developments
as they happen.

Aside, it’s also a problem we have with the OGC APIs, the yaml service files are pieced together
by hand and templated a bit to add a list of valid values, but no attempt is made to check if clients
can be generated, and if they would actually work. And again, same deal as above, we need more
people, or some sponsorship to figure out a yaml file generation from code, which would make it
happen almost effortlessly.

Best regards
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

5

yaml files created as documentation effort are yaml files of Swagger API 2.0 format, because it’s the only one that exists (with 3.0) and it has been created by Swagger, a standard de facto.
When yaml files correctly describe the REST services the server offers, the client that the Swagger generator produces is always working.

If a good client cannot be generated by Swagger generator tool from a yaml file that has a Swagger yaml format, it means that the content of the yaml file is wrong.
It’s what I’ve detected, and you can see it on https://docs.geoserver.org/stable/en/api/#1.0.0/workspaces.yaml swagger UI description for postWorkspaces method information where it says that workspaceBody is about to send a
{
“name” : “string”
}
when you know that the server is waiting for a :
{
“workspace”: {
“name”: “string”
}
}

(see attachment) and this is what I’ve corrected on a sample file (an attachment of my previous e-mail, I put corrected yaml file in this one too). Nothing else has to be done - yet - than correcting faulty yaml declarations, and this should not be a big problem.

For how to correct REST services in the long term and completely, and ensure their accuracy with server side about what is given as documentation and for generation, I agree with you. It’s another work.
For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

But summarizing :
I’ve corrected the faulty yaml file (for postWorkspaces method only, yet) and now it allows client generation by Swagger that works with Geoserver : geoNode project is not needed for calling it.

Workspace workspace = new Workspace();
workspace.setName(“ecoemploi”);
WorkspaceRequest workspaceRequest = new WorkspaceRequest();
workspaceRequest.setWorkspace(workspace);

WorkspacesApi workspacesApi = new WorkspacesApi(apiClient());
String response = workspacesApi.postWorkspaces(workspaceRequest, true);

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

Regards,

Marc.

(attachments)

geoserver_wrong_call.png
workspaces.yaml (8.64 KB)

···

On 07/02/2021 18:48, Andrea Aime wrote:

On Sun, Feb 7, 2021 at 6:10 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

I think no tool or no part of Geoserver use the content of the yaml rest
config files at the moment.

Correct, they have been hand-written as a documentation effort during the 2017 code sprint.
Generating a client was not a goal back then, it’s still not today (they are not really being maintained all that much).

There is also a separate issue that would need to be handled, that is, the REST representation
changes as soon as one of the internal config objects (CatalogInfo implementers) changes, which
happens every one and then, but without anyone ever caring to update the YAML files.
The changes are always made so that new fields are not mandatory, in order to allow backwards
compatibility with the old stored XML files.
Nothing in REST needs to be touched, as the representations are build using
a reflection like mechanism (XStream in particular), but a client written in a type safe language might
have issues adapting.

As far as I know, core devs are not generating clients, and my colleagues that are using the REST
API do so using a hand-written python library, https://github.com/GeoNode/geoserver-restconfig
I guess these clients are tolerant to representation changing,

I believe having YAML files that can be used to generate clients from is a worthy goal, but
we need more developers to get involved in the project and take care of that part, either

by generating the yamls from code (to keep the aligned at all time), or trying to follow the developments
as they happen.

Aside, it’s also a problem we have with the OGC APIs, the yaml service files are pieced together
by hand and templated a bit to add a list of valid values, but no attempt is made to check if clients
can be generated, and if they would actually work. And again, same deal as above, we need more
people, or some sponsorship to figure out a yaml file generation from code, which would make it
happen almost effortlessly.

Best regards
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

6

For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

I don’t see the need of a v2, you’re not changing the existing REST API implementation and the resource representations no? Just its description.
Nobody is using the yamls to generate clients right now, anyways.

Also, in general, versioning will likely not work, the API changes little by little as new needs pop up, we’d be at “v200” (exaggerating of course) if we considered every
model change happened so far, and every new resource addition or new request parameter.

What we typically try to do, is to preserve backwards compatibility, that is, a client written for a previous release should keep on working with the
new one (new fields are optional), new parameters in resource requests are optional.

As said in my previous mail, the yamls are just documentation at the moment: I don’t know why you explain that a client can be generated
from YAML files if they are correctly set up (we know that very well), as I told you already, right now it’s not a goal, due to the limited staffing of the project.
The yaml files have been hand written once, and then mostly left there to rot.

If you want to join in the effort to make the yaml files maintainable, and provide resources for both its initial setup and long term maintenance, then we can make
“yaml as code generation support” a project goal too. Otherwise there is nothing to discuss, it’s not a matter of “right or wrong”,
it’s a matter of having the man hours to do an initial version of it, and then look after it in the long term.
Existing developers are busy up to their eyeballs and more, there is no amount of reasoning that will change that.

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

First step, treat it as documentation and simply issue a fix for the yaml.

Second step, if you can write a test that generates a client, and can use it in an interaction with a GeoServer,
in a fully automated way, that is also welcomed (you might need to setup a GitHub action build too, if the
test needs a live GeoServer to work against, otherwise no one will run those tests).

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
gets a test failure) and you might not need to be involved long term all that much (but plan for some).

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

7

Ok. I stop considering the problem.

When you write : “Nobody is using the yamls to generate clients right now, anyways.”
I’m exactly. Because now many REST services are described by them and it’s more easy to have client generated by Swagger : it avoids constructing request and parsing json or xml response to put them in an (or some) objects manually.

And when I was searching for a solution on GIS Stackexchange, ignoring where the problem was coming from, you were among who told me to come on Geoserver project instead of asking, and handle it myself if I wanted a solution.

It’s what I did, I’ve spent twenty hours already on it and found one.

And now, you send me a different response.

Le lun. 8 févr. 2021 à 08:48, Andrea Aime <andrea.aime@anonymised.com> a écrit :

On Sun, Feb 7, 2021 at 10:00 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

I don’t see the need of a v2, you’re not changing the existing REST API implementation and the resource representations no? Just its description.
Nobody is using the yamls to generate clients right now, anyways.

Also, in general, versioning will likely not work, the API changes little by little as new needs pop up, we’d be at “v200” (exaggerating of course) if we considered every
model change happened so far, and every new resource addition or new request parameter.

What we typically try to do, is to preserve backwards compatibility, that is, a client written for a previous release should keep on working with the
new one (new fields are optional), new parameters in resource requests are optional.

As said in my previous mail, the yamls are just documentation at the moment: I don’t know why you explain that a client can be generated
from YAML files if they are correctly set up (we know that very well), as I told you already, right now it’s not a goal, due to the limited staffing of the project.
The yaml files have been hand written once, and then mostly left there to rot.

If you want to join in the effort to make the yaml files maintainable, and provide resources for both its initial setup and long term maintenance, then we can make
“yaml as code generation support” a project goal too. Otherwise there is nothing to discuss, it’s not a matter of “right or wrong”,
it’s a matter of having the man hours to do an initial version of it, and then look after it in the long term.
Existing developers are busy up to their eyeballs and more, there is no amount of reasoning that will change that.

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

First step, treat it as documentation and simply issue a fix for the yaml.

Second step, if you can write a test that generates a client, and can use it in an interaction with a GeoServer,
in a fully automated way, that is also welcomed (you might need to setup a GitHub action build too, if the
test needs a live GeoServer to work against, otherwise no one will run those tests).

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

8

Hi Marc,
I’m not sending you a different response, but it’s probably not what you wanted.

The three answers I gave you all point towards solving the problem, you just have to be the one making the PRs, as I said, they
would be welcomed. If your expectation is that others would stop what they are doing and help you, then indeed, you will not get that.

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

9

Hey, I just saw this, sorry for being late to the party

There’s a lot going on in this discussion, so I’ll try to be succinct.

First and foremost, I am generating a Java client from the OpenAPI documents. Not the swagger 2 ones provided as documentation, but OAS3 ones created using those as reference.

Here’s a project I just made public a couple of days ago: https://github.com/camptocamp/geoserver-rest-openapi

At camptocamp we’re using it in production for over a year in an internal project, but now I moved it as a standalone project to be used by the OSS project geOrchestra.

It only implements the minimum I needed for that internal project so far. That is, working with workspaces, datastores, feature types, layers, and styles. And probably not even all of it.

Also, it focuses on JSON representation only.

On the bright side, it’s a standalone library, uses the OpenAPI maven code generator, and runs integration tests against a GeoServer docker container managed by the maven life cycle.

CI builds are set up using github workflows: https://github.com/camptocamp/geoserver-rest-openapi/actions

···

Gabriel Roldán

10

On Wed, Feb 10, 2021 at 1:58 AM Gabriel Roldan <gabriel.roldan@anonymised.com> wrote:

First and foremost, I am generating a Java client from the OpenAPI documents. Not the swagger 2 ones provided as documentation, but OAS3 ones created using those as reference.

Here’s a project I just made public a couple of days ago: https://github.com/camptocamp/geoserver-rest-openapi

At camptocamp we’re using it in production for over a year in an internal project, but now I moved it as a standalone project to be used by the OSS project geOrchestra.

It only implements the minimum I needed for that internal project so far. That is, working with workspaces, datastores, feature types, layers, and styles. And probably not even all of it.

Also, it focuses on JSON representation only.

Nice to see a Java based client, geoserver-manager died several years ago leaving no replacement as a Java based GeoServer REST client.
If enough people are interested, and it lives long enough, no doubt it will eventually cover fully functionality, or close to.

That said, writing the OAS3 files and the generated client wrapper code is hard, often having to debug GeoServer itself to figure out what’s going on in the server to be able of mimicking it on the api definition. There are several inconsistencies, not only in the outdated/incomplete swagger2 files, but in the server itself, most of which I think are due to using XStream to generate the JSON representations, like a single object instead of an array when an array is expected but the result contains a single element, excessive wrapping of objects, and several other annoyances I didn’t care to document properly at the time, but believe me, I thought having a working set of OAS3 specs would be a nice first step towards defining a v2 api with clearly defined api contracts and code-generated server stubs.

Yes, we are painfully aware of all the above.

Achieving that would of course require significant resources so it most probably will die in the wish-list.

Indeed, a fresh start for REST services would require solid funding and a new dedicated maintainer for it.
The existing REST API needs close to no maintenance, besides the bug fixing and new functionality, which are both directly funded, when happening,
Also the existing clients are working good enough for many, so the two REST APIs would need to be kept alive for a few years,
giving existing deployers time to migrate.

That’s the main crux of all this discussion, there is a big chasm between those that like different, newer, cleaner functionality,
but have nothing to offer in terms of funds or development time, and those that are trying to keep the project alive, already
fully busy with paid work against the project, and already having a working solution for the issue at hand.

I see kind of the same issue happening down in OGC API modules, they are moving one testbed or code sprint at a time, but there
is very little business around them (mostly research projects), cause they are providing a different solution to a problem that
was solved already 20 years ago, and for which many have working clients. Those that don’t are clamoring for the new API,
sure, but also providing no resources to make the modules go up and graduate.

I’m just glad despite all the noise, we are managing to keep the project alive and going, pruning dead branches and moving on,
instead of living the current situation in GDAL, where the project maintainer seems to have had enough of all the external requests and pressure,
and disappeared from both the mailing list and twitter (I surely hope it’s just a vacation… we’ll see).

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

11

Hello,

Your tool shows the need to have a way to automate installation(s) of Geoserver sometimes. And I have it too.
curl statements aren’t enough for this task, and OpenAPI is a convenient way to drive its installation. Using a client API to override it like you do and offer additional help is useful (I have to create internally mine too !). But the beginning is that OpenAPI calls have to work, and there’s some corrections to do in yaml files provided, for that.

Currently, in [GEOS-9886] issue (that I’ve renamed to focus only on workspaces.yaml content), I’ve addressed three flaws :

  • One in the input content sent to postWorkspaces, that doesn’t send the JSON the server expects.
  • Another in the output of geoserver/rest/workspaces (= list all the workspaces) that should expect an array incoming in return of a call, instead of an object.
  • Another in the output of geoserver/rest/workspaces/{workspaceName} (= a single workspace) that should expect a “workspace:” incoming instead of an anonymous object.

I still have to test deleteWorkspace method, and then the workspaces.yaml file will be corrected.

I came too far with my idea of a V2 rest yaml files describing the Geoserver REST services, using annotations on server classes to generate these yaml files and ensure their accuracy,
but the problem is established : if on that first workspaces.yaml file I’ve found three bugs, the other might have the same amount and, one by one, depending on my needs, I will have to check and correct them.

Regards,

Marc.

Le mer. 10 févr. 2021 à 01:58, Gabriel Roldan <gabriel.roldan@anonymised.com> a écrit :

Hey, I just saw this, sorry for being late to the party

There’s a lot going on in this discussion, so I’ll try to be succinct.

First and foremost, I am generating a Java client from the OpenAPI documents. Not the swagger 2 ones provided as documentation, but OAS3 ones created using those as reference.

Here’s a project I just made public a couple of days ago: https://github.com/camptocamp/geoserver-rest-openapi

At camptocamp we’re using it in production for over a year in an internal project, but now I moved it as a standalone project to be used by the OSS project geOrchestra.

It only implements the minimum I needed for that internal project so far. That is, working with workspaces, datastores, feature types, layers, and styles. And probably not even all of it.

Also, it focuses on JSON representation only.

On the bright side, it’s a standalone library, uses the OpenAPI maven code generator, and runs integration tests against a GeoServer docker container managed by the maven life cycle.

CI builds are set up using github workflows: https://github.com/camptocamp/geoserver-rest-openapi/actions

That said, writing the OAS3 files and the generated client wrapper code is hard, often having to debug GeoServer itself to figure out what's going on in the server to be able of mimicking it on the api definition. There are several inconsistencies, not only in the outdated/incomplete swagger2 files, but in the server itself, most of which I think are due to using XStream to generate the JSON representations, like a single object instead of an array when an array is expected but the result contains a single element, excessive wrapping of objects, and several other annoyances I didn't care to document properly at the time, but believe me, I thought having a working set of OAS3 specs would be a nice first step towards defining a v2 api with clearly defined api contracts and code-generated server stubs. Achieving that would of course require significant resources so it most probably will die in the wish-list.

But by all means, feel free to check whether that project is of use to you and to contribute to it. Given enough community interest, we could even manage to land it as part of geoserver’s codebase.

This Andrea’s comment is of most interest and I think I know how to address it

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
> gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Since the OAS3 api object model (catalog info objects, etc) mirrors GeoServer Catalog object model, I’ve experience using mapstruct to create code-generated object model mappers, and to make the code generation fail if something changes, which is a good way to ensure both stay in sync. That’d be of special interest in case a v2 api would be defined, but I had the intention of creating those mappers nonetheless at least for the sake of keeping the models in sync.

As a final note, if I had the resources, I’d very much would like to implement such new api version, but would definitely do so as a microservice in the context of this project https://github.com/camptocamp/geoserver-microservices

Hope that helps,

Gabriel

On Mon, 8 Feb 2021 at 04:49, Andrea Aime <andrea.aime@anonymised.com…> wrote:

On Sun, Feb 7, 2021 at 10:00 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

I don’t see the need of a v2, you’re not changing the existing REST API implementation and the resource representations no? Just its description.
Nobody is using the yamls to generate clients right now, anyways.

Also, in general, versioning will likely not work, the API changes little by little as new needs pop up, we’d be at “v200” (exaggerating of course) if we considered every
model change happened so far, and every new resource addition or new request parameter.

What we typically try to do, is to preserve backwards compatibility, that is, a client written for a previous release should keep on working with the
new one (new fields are optional), new parameters in resource requests are optional.

As said in my previous mail, the yamls are just documentation at the moment: I don’t know why you explain that a client can be generated
from YAML files if they are correctly set up (we know that very well), as I told you already, right now it’s not a goal, due to the limited staffing of the project.
The yaml files have been hand written once, and then mostly left there to rot.

If you want to join in the effort to make the yaml files maintainable, and provide resources for both its initial setup and long term maintenance, then we can make
“yaml as code generation support” a project goal too. Otherwise there is nothing to discuss, it’s not a matter of “right or wrong”,
it’s a matter of having the man hours to do an initial version of it, and then look after it in the long term.
Existing developers are busy up to their eyeballs and more, there is no amount of reasoning that will change that.

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

First step, treat it as documentation and simply issue a fix for the yaml.

Second step, if you can write a test that generates a client, and can use it in an interaction with a GeoServer,
in a fully automated way, that is also welcomed (you might need to setup a GitHub action build too, if the
test needs a live GeoServer to work against, otherwise no one will run those tests).

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Gabriel Roldán

12

If you or Gabriel want to update the yaml files with fixed versions, please issue a PR.

I believe no one has a contract to work on that during working hours, but we reserve a few hours a week to review PRs.

A detailed ticket is a good thing, mind, but it’s very unlikely to spur any actual action.

Cheers
Andrea

···

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

13

Yep, spot on.

···

Gabriel Roldán

14

Hello,

Your tool shows the need to have a way to automate installation(s) of Geoserver sometimes. And I have it too.
curl statements aren’t enough for this task, and OpenAPI is a convenient way to drive its installation. Using a client API to override it like you do and offer additional help is useful (I have to create internally mine too !). But the beginning is that OpenAPI calls have to work, and there’s some corrections to do in yaml files provided, for that.

Currently, in [GEOS-9886] issue (that I’ve renamed to focus only on workspaces.yaml content), I’ve addressed three flaws :

  • One in the input content sent to postWorkspaces, that doesn’t send the JSON the server expects.
  • Another in the output of geoserver/rest/workspaces (= list all the workspaces) that should expect an array incoming in return of a call, instead of an object.
  • Another in the output of geoserver/rest/workspaces/{workspaceName} (= a single workspace) that should expect a “workspace:” incoming instead of an anonymous object.

If you want to fix the swagger 2 files go for it. I was just pointing out there’re those OAS3 ones with the errors you mention already fixed if you want to use them instead (i.e. given swagger 2 is superseded by OAS3).

···

Gabriel Roldán

15

Oh, Ok. Sorry my english isn’t so good and I hadn’t understood all.
An OpenAPI 3 version of geoserver REST contracts already exist and is debugged ?
It’s a very good news for me and there’s absolutely no need to continue using Swagger V2 yaml, then : you’re right.
But where are these OAS 3 files ? I’d better use them immediately.

Regards,

Marc.

Le mer. 10 févr. 2021 à 15:30, Gabriel Roldan <gabriel.roldan@anonymised.com> a écrit :

On Wed, 10 Feb 2021 at 07:57, Marc LE BIHAN <mlebihan29@anonymised.com> wrote:

Hello,

Your tool shows the need to have a way to automate installation(s) of Geoserver sometimes. And I have it too.
curl statements aren’t enough for this task, and OpenAPI is a convenient way to drive its installation. Using a client API to override it like you do and offer additional help is useful (I have to create internally mine too !). But the beginning is that OpenAPI calls have to work, and there’s some corrections to do in yaml files provided, for that.

Currently, in [GEOS-9886] issue (that I’ve renamed to focus only on workspaces.yaml content), I’ve addressed three flaws :

  • One in the input content sent to postWorkspaces, that doesn’t send the JSON the server expects.
  • Another in the output of geoserver/rest/workspaces (= list all the workspaces) that should expect an array incoming in return of a call, instead of an object.
  • Another in the output of geoserver/rest/workspaces/{workspaceName} (= a single workspace) that should expect a “workspace:” incoming instead of an anonymous object.

If you want to fix the swagger 2 files go for it. I was just pointing out there’re those OAS3 ones with the errors you mention already fixed if you want to use them instead (i.e. given swagger 2 is superseded by OAS3).

I still have to test deleteWorkspace method, and then the workspaces.yaml file will be corrected.

I came too far with my idea of a V2 rest yaml files describing the Geoserver REST services, using annotations on server classes to generate these yaml files and ensure their accuracy,
but the problem is established : if on that first workspaces.yaml file I’ve found three bugs, the other might have the same amount and, one by one, depending on my needs, I will have to check and correct them.

Regards,

Marc.

Le mer. 10 févr. 2021 à 01:58, Gabriel Roldan <gabriel.roldan@anonymised.com> a écrit :

Hey, I just saw this, sorry for being late to the party

There’s a lot going on in this discussion, so I’ll try to be succinct.

First and foremost, I am generating a Java client from the OpenAPI documents. Not the swagger 2 ones provided as documentation, but OAS3 ones created using those as reference.

Here’s a project I just made public a couple of days ago: https://github.com/camptocamp/geoserver-rest-openapi

At camptocamp we’re using it in production for over a year in an internal project, but now I moved it as a standalone project to be used by the OSS project geOrchestra.

It only implements the minimum I needed for that internal project so far. That is, working with workspaces, datastores, feature types, layers, and styles. And probably not even all of it.

Also, it focuses on JSON representation only.

On the bright side, it’s a standalone library, uses the OpenAPI maven code generator, and runs integration tests against a GeoServer docker container managed by the maven life cycle.

CI builds are set up using github workflows: https://github.com/camptocamp/geoserver-rest-openapi/actions

That said, writing the OAS3 files and the generated client wrapper code is hard, often having to debug GeoServer itself to figure out what's going on in the server to be able of mimicking it on the api definition. There are several inconsistencies, not only in the outdated/incomplete swagger2 files, but in the server itself, most of which I think are due to using XStream to generate the JSON representations, like a single object instead of an array when an array is expected but the result contains a single element, excessive wrapping of objects, and several other annoyances I didn't care to document properly at the time, but believe me, I thought having a working set of OAS3 specs would be a nice first step towards defining a v2 api with clearly defined api contracts and code-generated server stubs. Achieving that would of course require significant resources so it most probably will die in the wish-list.

But by all means, feel free to check whether that project is of use to you and to contribute to it. Given enough community interest, we could even manage to land it as part of geoserver’s codebase.

This Andrea’s comment is of most interest and I think I know how to address it

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
> gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Since the OAS3 api object model (catalog info objects, etc) mirrors GeoServer Catalog object model, I’ve experience using mapstruct to create code-generated object model mappers, and to make the code generation fail if something changes, which is a good way to ensure both stay in sync. That’d be of special interest in case a v2 api would be defined, but I had the intention of creating those mappers nonetheless at least for the sake of keeping the models in sync.

As a final note, if I had the resources, I’d very much would like to implement such new api version, but would definitely do so as a microservice in the context of this project https://github.com/camptocamp/geoserver-microservices

Hope that helps,

Gabriel

On Mon, 8 Feb 2021 at 04:49, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Feb 7, 2021 at 10:00 PM Marc Le Bihan <mlebihan29@…403…> wrote:

For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

I don’t see the need of a v2, you’re not changing the existing REST API implementation and the resource representations no? Just its description.
Nobody is using the yamls to generate clients right now, anyways.

Also, in general, versioning will likely not work, the API changes little by little as new needs pop up, we’d be at “v200” (exaggerating of course) if we considered every
model change happened so far, and every new resource addition or new request parameter.

What we typically try to do, is to preserve backwards compatibility, that is, a client written for a previous release should keep on working with the
new one (new fields are optional), new parameters in resource requests are optional.

As said in my previous mail, the yamls are just documentation at the moment: I don’t know why you explain that a client can be generated
from YAML files if they are correctly set up (we know that very well), as I told you already, right now it’s not a goal, due to the limited staffing of the project.
The yaml files have been hand written once, and then mostly left there to rot.

If you want to join in the effort to make the yaml files maintainable, and provide resources for both its initial setup and long term maintenance, then we can make
“yaml as code generation support” a project goal too. Otherwise there is nothing to discuss, it’s not a matter of “right or wrong”,
it’s a matter of having the man hours to do an initial version of it, and then look after it in the long term.
Existing developers are busy up to their eyeballs and more, there is no amount of reasoning that will change that.

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

First step, treat it as documentation and simply issue a fix for the yaml.

Second step, if you can write a test that generates a client, and can use it in an interaction with a GeoServer,
in a fully automated way, that is also welcomed (you might need to setup a GitHub action build too, if the
test needs a live GeoServer to work against, otherwise no one will run those tests).

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Gabriel Roldán

–

Gabriel Roldán

16

Oh, Ok. Sorry my english isn’t so good and I hadn’t understood all.

np

An OpenAPI 3 version of geoserver REST contracts already exist and is debugged ?
It’s a very good news for me and there’s absolutely no need to continue using Swagger V2 yaml, then : you’re right.
But where are these OAS 3 files ? I’d better use them immediately.

https://github.com/camptocamp/geoserver-rest-openapi/tree/master/openapi/1.0.0

Note the catalog object model is in catalog.yml, and workspaces.yml uses it. The code generation configured in that project’s poms takes care of it. Either use it as it (the client it generates) or use it as a guideline.

The integration tests could be of help https://github.com/camptocamp/geoserver-rest-openapi/tree/master/java-client/src/test/java/org/geoserver/restconfig/client

Cheers,

···

Gabriel Roldán

17

I see : you interfaced or skipped Geoserver flawed 2.0 yaml and produced a working OAS 3.0 version instead.
That’s an impressive work !

Well, I have to think.
Knowing that yaml (Swagger 2.0) descriptions of Geoserver config REST services are wrong, it disturbs me to see them remaining online,
making people believe that they can trust them either for documentation or generation, and that isn’t the case. It’s like you offer SOAP web-services with a wrong WSDL file coming along…

I would like to see what happens if Geoserver REST config server attempts to generate itself the json description (or yaml one) that depict itsellf the best, using springdoc-openapi-maven-plugin:generate.

···

On 10/02/2021 17:50, Gabriel Roldan wrote:

On Wed, 10 Feb 2021 at 12:21, Marc LE BIHAN <mlebihan29@anonymised.com> wrote:

Oh, Ok. Sorry my english isn’t so good and I hadn’t understood all.

np

An OpenAPI 3 version of geoserver REST contracts already exist and is debugged ?
It’s a very good news for me and there’s absolutely no need to continue using Swagger V2 yaml, then : you’re right.
But where are these OAS 3 files ? I’d better use them immediately.

https://github.com/camptocamp/geoserver-rest-openapi/tree/master/openapi/1.0.0

Note the catalog object model is in catalog.yml, and workspaces.yml uses it. The code generation configured in that project’s poms takes care of it. Either use it as it (the client it generates) or use it as a guideline.

The integration tests could be of help https://github.com/camptocamp/geoserver-rest-openapi/tree/master/java-client/src/test/java/org/geoserver/restconfig/client

Cheers,

Regards,

Marc.

Le mer. 10 févr. 2021 à 15:30, Gabriel Roldan <gabriel.roldan@anonymised.com> a écrit :

On Wed, 10 Feb 2021 at 07:57, Marc LE BIHAN <mlebihan29@anonymised.com> wrote:

Hello,

Your tool shows the need to have a way to automate installation(s) of Geoserver sometimes. And I have it too.
curl statements aren’t enough for this task, and OpenAPI is a convenient way to drive its installation. Using a client API to override it like you do and offer additional help is useful (I have to create internally mine too !). But the beginning is that OpenAPI calls have to work, and there’s some corrections to do in yaml files provided, for that.

Currently, in [GEOS-9886] issue (that I’ve renamed to focus only on workspaces.yaml content), I’ve addressed three flaws :

  • One in the input content sent to postWorkspaces, that doesn’t send the JSON the server expects.
  • Another in the output of geoserver/rest/workspaces (= list all the workspaces) that should expect an array incoming in return of a call, instead of an object.
  • Another in the output of geoserver/rest/workspaces/{workspaceName} (= a single workspace) that should expect a “workspace:” incoming instead of an anonymous object.

If you want to fix the swagger 2 files go for it. I was just pointing out there’re those OAS3 ones with the errors you mention already fixed if you want to use them instead (i.e. given swagger 2 is superseded by OAS3).

I still have to test deleteWorkspace method, and then the workspaces.yaml file will be corrected.

I came too far with my idea of a V2 rest yaml files describing the Geoserver REST services, using annotations on server classes to generate these yaml files and ensure their accuracy,
but the problem is established : if on that first workspaces.yaml file I’ve found three bugs, the other might have the same amount and, one by one, depending on my needs, I will have to check and correct them.

Regards,

Marc.

Le mer. 10 févr. 2021 à 01:58, Gabriel Roldan <gabriel.roldan@anonymised.com> a écrit :

Hey, I just saw this, sorry for being late to the party

There’s a lot going on in this discussion, so I’ll try to be succinct.

First and foremost, I am generating a Java client from the OpenAPI documents. Not the swagger 2 ones provided as documentation, but OAS3 ones created using those as reference.

Here’s a project I just made public a couple of days ago: https://github.com/camptocamp/geoserver-rest-openapi

At camptocamp we’re using it in production for over a year in an internal project, but now I moved it as a standalone project to be used by the OSS project geOrchestra.

It only implements the minimum I needed for that internal project so far. That is, working with workspaces, datastores, feature types, layers, and styles. And probably not even all of it.

Also, it focuses on JSON representation only.

On the bright side, it’s a standalone library, uses the OpenAPI maven code generator, and runs integration tests against a GeoServer docker container managed by the maven life cycle.

CI builds are set up using github workflows: https://github.com/camptocamp/geoserver-rest-openapi/actions

That said, writing the OAS3 files and the generated client wrapper code is hard, often having to debug GeoServer itself to figure out what's going on in the server to be able of mimicking it on the api definition. There are several inconsistencies, not only in the outdated/incomplete swagger2 files, but in the server itself, most of which I think are due to using XStream to generate the JSON representations, like a single object instead of an array when an array is expected but the result contains a single element, excessive wrapping of objects, and several other annoyances I didn't care to document properly at the time, but believe me, I thought having a working set of OAS3 specs would be a nice first step towards defining a v2 api with clearly defined api contracts and code-generated server stubs. Achieving that would of course require significant resources so it most probably will die in the wish-list.

But by all means, feel free to check whether that project is of use to you and to contribute to it. Given enough community interest, we could even manage to land it as part of geoserver’s codebase.

This Andrea’s comment is of most interest and I think I know how to address it

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
> gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Since the OAS3 api object model (catalog info objects, etc) mirrors GeoServer Catalog object model, I’ve experience using mapstruct to create code-generated object model mappers, and to make the code generation fail if something changes, which is a good way to ensure both stay in sync. That’d be of special interest in case a v2 api would be defined, but I had the intention of creating those mappers nonetheless at least for the sake of keeping the models in sync.

As a final note, if I had the resources, I’d very much would like to implement such new api version, but would definitely do so as a microservice in the context of this project https://github.com/camptocamp/geoserver-microservices

Hope that helps,

Gabriel

On Mon, 8 Feb 2021 at 04:49, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Sun, Feb 7, 2021 at 10:00 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

For that, I think the best thing to do would be to annotate server classes and object to allow Swagger/Open Api to produce their exact declarations of services in a new yaml file.
But this new yaml file and Swagger-UI documentation should be in another URL, let say /geoserver/rest/v2/… to ensure that existing code that rely on version 1 of the API can continue to use /geoserver/rest/… without trouble .
(but this is only the beginning of a solution…)

I don’t see the need of a v2, you’re not changing the existing REST API implementation and the resource representations no? Just its description.
Nobody is using the yamls to generate clients right now, anyways.

Also, in general, versioning will likely not work, the API changes little by little as new needs pop up, we’d be at “v200” (exaggerating of course) if we considered every
model change happened so far, and every new resource addition or new request parameter.

What we typically try to do, is to preserve backwards compatibility, that is, a client written for a previous release should keep on working with the
new one (new fields are optional), new parameters in resource requests are optional.

As said in my previous mail, the yamls are just documentation at the moment: I don’t know why you explain that a client can be generated
from YAML files if they are correctly set up (we know that very well), as I told you already, right now it’s not a goal, due to the limited staffing of the project.
The yaml files have been hand written once, and then mostly left there to rot.

If you want to join in the effort to make the yaml files maintainable, and provide resources for both its initial setup and long term maintenance, then we can make
“yaml as code generation support” a project goal too. Otherwise there is nothing to discuss, it’s not a matter of “right or wrong”,
it’s a matter of having the man hours to do an initial version of it, and then look after it in the long term.
Existing developers are busy up to their eyeballs and more, there is no amount of reasoning that will change that.

My question is :
Do you want this file to be integrated in the restconfig project with a test if I can create one that is convincing,
or do you think it’s too much, too early, and I better keep these corrected file for myself, for my own use only ?

First step, treat it as documentation and simply issue a fix for the yaml.

Second step, if you can write a test that generates a client, and can use it in an interaction with a GeoServer,
in a fully automated way, that is also welcomed (you might need to setup a GitHub action build too, if the
test needs a live GeoServer to work against, otherwise no one will run those tests).

Third, if you want to have reliable yamls that you can generate clients for, make a plan for it, write a GSIP,
resource it fully, and then implement it. Do it in a way that the system is self sustaining (every deviation
gets a test failure) and you might not need to be involved long term all that much (but plan for some).

Regards, Andrea Aime

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Gabriel Roldán

–

Gabriel Roldán

–

Gabriel Roldán

18

Good point, I’ve made a PR warning potential users that the files are not usable for client generation, and generally outdated:
https://github.com/geoserver/geoserver/pull/4717

We can remove it if/when someone steps up to fix them.

Cheers
Andrea

···

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

19

I’m working to correct some, but what Gabriel useful work makes me think about, is that rather than correcting 2.0 yaml version files, it’s more clever to produce immediately correct OpenAPI 3 compatible ones.
And my idea is that the Geoserver rest-config project should be able to produce them itself, provided it is guided with OpenAPI documentation annotations to preserve all the documentation work done before in the yaml files. I’m making attempts, these days.

If I can’t, I will PR corrected 2.0 yaml files only, but this when I will have checked and ensured that :

  1. Their declared input and output (that Swagger-UI detects from the yaml content) match the examples humans wrote in that yaml : it isn’t the case today.
  2. A generated client is able to make all methods working in a simple test (creating, removing, listing workspaces, for example).

Le jeu. 11 févr. 2021 à 11:03, Andrea Aime <andrea.aime@anonymised.com> a écrit :

On Wed, Feb 10, 2021 at 7:07 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

Knowing that yaml (Swagger 2.0) descriptions of Geoserver config REST services are wrong, it disturbs me to see them remaining online,
making people believe that they can trust them either for documentation or generation, and that isn’t the case. It’s like you offer SOAP web-services with a wrong WSDL file coming along…

Good point, I’ve made a PR warning potential users that the files are not usable for client generation, and generally outdated:
https://github.com/geoserver/geoserver/pull/4717

We can remove it if/when someone steps up to fix them.

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

20

Hello,

About workspaces.yaml :I added the corrected yaml file and a test for

[GEOS-9886] workspaces.yaml doesn’t carry correct descriptions of REST services it presents. (as an effect, a working Swagger client cannot be generated for it)

Before creating a pull request to send the corrected geoserver/doc/en/api/1.0.0/workspaces.yaml,
I would like to find a way to put the test I did in geoserver. But it could a long work, because I don’t know how to reproduce all the steps of my own test project in a manner compatible to geoserver project rules.
And I would like to check the next point to see if it can offer a better solution too.

About openapi.json generation from existing REST servicesWhen adding this dependency to an application (I’ve used it’s version 1.5.3, compatible with Spring-boot 2.4.1)

org.springdoc springdoc-openapi-ui

Swagger/OpenAPI immediately makes available

http://localhost:8080/swagger-ui.html
that produces the same web interface than the one on Geoserver documentation website.

http://localhost:8080/v3/api-docs
that describes the REST services in a json format (instead of a yaml one, but I don’t know if it matters).

This path can be changed with this property :
springdoc.api-docs.path=/geoserver/rest/v3/api-docs

But to see it if works, I’m lacking Geoserver project knowledge :
For the time of testing (at home only), to see what OpenAPI produces from Geoserver REST classes,

  1. May I (temporarily) add a security rule to authorize path /v3/api-docs, allowing direct access to http://localhost:8080/v3/api-docs, and how ?

  2. Else may I declare that property, and then : where, to make it taken into account ?
    springdoc.api-docs.path=/geoserver/rest/v3/api-docs

Thanks,

Marc Le Bihan

···

On 11/02/2021 16:10, Marc LE BIHAN wrote:

I’m working to correct some, but what Gabriel useful work makes me think about, is that rather than correcting 2.0 yaml version files, it’s more clever to produce immediately correct OpenAPI 3 compatible ones.
And my idea is that the Geoserver rest-config project should be able to produce them itself, provided it is guided with OpenAPI documentation annotations to preserve all the documentation work done before in the yaml files. I’m making attempts, these days.

If I can’t, I will PR corrected 2.0 yaml files only, but this when I will have checked and ensured that :

  1. Their declared input and output (that Swagger-UI detects from the yaml content) match the examples humans wrote in that yaml : it isn’t the case today.
  2. A generated client is able to make all methods working in a simple test (creating, removing, listing workspaces, for example).

Le jeu. 11 févr. 2021 à 11:03, Andrea Aime <andrea.aime@anonymised.com> a écrit :

On Wed, Feb 10, 2021 at 7:07 PM Marc Le Bihan <mlebihan29@anonymised.com> wrote:

Knowing that yaml (Swagger 2.0) descriptions of Geoserver config REST services are wrong, it disturbs me to see them remaining online,
making people believe that they can trust them either for documentation or generation, and that isn’t the case. It’s like you offer SOAP web-services with a wrong WSDL file coming along…

Good point, I’ve made a PR warning potential users that the files are not usable for client generation, and generally outdated:
https://github.com/geoserver/geoserver/pull/4717

We can remove it if/when someone steps up to fix them.

Cheers
Andrea

== GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.