Hi all,
Camptocamp is interested in contributing to graduate the gs-ogcapi-features module (and by extension gs-ogcapi-core) to supported status.
I’ll be getting used to it, but wonder if that’s an achievable goal and if you have any clue of what it would entail.
Cheers,
Gabe
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Hi Gabriel,
as a goal, it is achievable, but there are a few roadblocks that need to be removed.
On the good side of things, we have a service that, last time I checked, managed to pass the
CITE tests for OGC API - Core and OGC API Features - Coordinate Reference Systems by Reference.
However, there are also issues that stem either from the nature of the OGC APIs and part from
the code being developed mostly as an accumulation of code sprints.
The issue with OGC APIs development is that an implementation of a service is an amalgamation
of various small specs, all of which are in different stages of development, some stable, other subject
to change.
The issue becomes evident when this is coupled with Spring MVC controllers, e.g. the items resource:
in the same method signature we have bits that are finalized (bbox, limit, datetime), bit that are
only formally finalized (filter, filter-lang, filter-crs) and others that are coming straight out of other
specifications (sortby).
The filter bit needs some clarification. Filter comes from Part 3, which defines a general filtering mechanism,
but the preferred filtering language is CQL2. I hear that CQL2 is nearing completion, but our implementation
for cql-text and cql-json (found in GeoTools) are several months old.
Now, in GeoServer a supported module is stable and reliable, spec wise: we cannot have someone implement
a client against a supported module, and then a few months down the line, maybe to grab a security fix,
have it stop working in a permanent way because the specification changed.
Long story short, an extension module should implement a stable specification.
So, we’ll have a need to separate OGC API Features so that the unstable bits are left out in the community, as
plugins, while the stable bits are found in the extension. To do this, you’ll have to go thought the service
implementation with a fine tooth comb and separate the various bits, opening extension points for the current
functionality that does not belong in extension.
Thankfully, filtering languages are pluggable already, although I’m not sure how to indicate that the default
is going to be something else (e.g. ECQL) of it going this way is even desirable, as at one point that default will
have to change.
The other troublesome bit is that the some of the core integration of the dispatcher has overridden some
Spring bits in a deep way to get support for the @anonymised.com annotations and
the general output format management (e.g., the support of “f=json”, or the variable interpretation of
what “f=json” means for example). I don’t see it a priority, but you might think otherwise.
The last bit is that this is a true community module, in that while I’ve been doing quite a bit of work on it,
a number of people have already contributed to it, so far with little friction. I also know that GeoCat is interested
in working on it (and already has, for example, contributed various bits to it) and also has some objectives to
achieve, I’ll let Jody explain more.
Generally speaking, graduating this module will have to be an effort that keeps everyone happy, so collaboration
and openness to compromise are going to be key here.
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Hi Gabe,
GeoCat is interested in doing so also - so I really wish to be sure to communicate and not trip on each other.
While Andrea has gone into some technical gotcha’s I have some user experience challenges to go over also.
Disclaimer that I respect the work done thus far; and many of the surprises for me come from starting to learn these new standards.
conformances
My customers really want fine grain control over the conformances; the ability to enable/disable on a case-by-case basis. This is going to be required as different regions have different mandatory requirements as we saw during the work done last year. It is also advantageous to limit the attack surface and only enable what is needed. This also provide an approach for all the optional not quite completed yet standards; they can be disabled by default?
user experience
I like the story of configure your layers in geoserver once and they will show up in additional services / protocols as they become available. Kind of the point of GSIP-202.
In https://github.com/geoserver/geoserver/pulls you can see I had some surprises integrating service descriptors in line with GSIP-202. I had fully expected to rename the WFS → Features and have the different protocols (OWS-WFS, OGCAPI-Features) be listed under this common heading. I could see changing the menu heading to “Features” with tabs for each service for example …
However each OGCAPI service has its own ServiceInfo with title/description - even if it does not provide any additional configuration over and above the open web service base (OGCAPI-Featuers uses WFSInfo for example). This was not what I expected so I am not sure I have a good idea how to proceed yet. David added some more “hints” to slot things under the correct heading; and there is not any way to configure the title / heading for OGCAPI-Features separately yet…
I am also not sure what to do with things like DIGGS? So while Andrea identifies some challenges integrating with Dispatcher; I want to highlight that we have a challenge integrating with the application experience also.
Per service collections vs everything collections
This is another fundamental how should it work question. If you look at our services each has been done independently with its own collects list. The other implementations (for example pygeopai) have a single collections list; which has more functionality dumped into it to reflect each ogcapi-service conformance enabled …
The “everything” collections is a much better user experience. List stuff and how to interact with it in one place. However it has its limits; even pygeoapi has ended up with “collection” and “record-collection” now…
Keeping them independent results in a service that is small enough we could test openapi client generation (but perhaps only the two of us share that dream eh?).
Aside: It looks like some things like ogcapi-records search are designed to be additions to all collections end-points (adding text search for example).
If we could answer this a bit better we could determine if a refactor / merge is needed?
Tech preview
Last year I relaxed Docker restriction to allow for nightly builds and public testing of OGCAPI services. Even putting that example in the user guide for cut and paste ease. I am not sure if it has resulted in any additional feedback/development/funding?
My other idea is to put up a “tech preview” of GeoServer only with OGCAPI services with a warning that “contains community components that have not been reviewed”. Similar to how Eclipse Foundation includes a warning on downloads that include incubating components…
Goal: Get more eyes on the OGCAPI protocols and thus more interest both development and funding …
Caution: This may not be worth it give our poor community response to participation. I have been disappointed with the response thus the 2024 roadmap challenges. It may be easier to interest people in new functionality; but we do need to look to sustainability also.
I am going to stop now …
–
Jody Garnett
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Sorry gabe, here is the correct PR: https://github.com/geoserver/geoserver/pull/7716 showing some of the user interface integration challenges
–
Jody Garnett
–
Jody Garnett
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Hi all,
sorry for the late reply, FOSS4G EU preparations kept me from following up on this one for a bit.
Observations below.
conformances
My customers really want fine grain control over the conformances; the ability to enable/disable on a case-by-case basis. This is going to be required as different regions have different mandatory requirements as we saw during the work done last year. It is also advantageous to limit the attack surface and only enable what is needed. This also provide an approach for all the optional not quite completed yet standards; they can be disabled by default?
That’s an interesting challenge… we’d have to be able to selectively disable resources, resource methods, and resource query parameters/body elements (some API do use JSON document as POST, e.g. STAC API search endpoint). At the same time, they should behave like they are not implemented, so not denied, but ignored: we cannot go out with an error message because you’re trying to use sortby and it’s been disabled, it should just be ignored as if the server did not implement it.
The above would make for a fair bit of hairy code, if implemented in the service controller directly, and some callbacks controlling how the request parsing is done, otherwise (pluggable and external, but still one class or one method per configurable behavior).
And oh, also plugins to edit the API document in order to remove the classes we do not want to expose.
Another possible take is to split down the module into many smaller parts (e.g., gs-ogcapi-features-sorting), that you just don’t install… but it would not work if you are doing a managed multi-tenant deploy.
user experience
I like the story of configure your layers in geoserver once and they will show up in additional services / protocols as they become available. Kind of the point of GSIP-202.
I’m confused… “configure your layers in geoserver once and they will show up in all compatible services” is what GeoServer has done since day one. GSIP-202 moved the OGC API landing pages in the same place as the capabilities documents for the same type of OWS service, which is very sensible, but unrelated to which services publish which layer (as you said, defaults to all layers, but can now be twaked at a layer level to choose a subset of them).
In https://github.com/geoserver/geoserver/pulls you can see I had some surprises integrating service descriptors in line with GSIP-202. I had fully expected to rename the WFS → Features and have the different protocols (OWS-WFS, OGCAPI-Features) be listed under this common heading. I could see changing the menu heading to “Features” with tabs for each service for example …
… the project is open source, don’t “expect”, just check the code 
I am also not sure what to do with things like DIGGS?
DGGS, Discrete Global Grid System. It does not fit with the others. If you add “STAC” it would not fit either, although one
might create a “catalog” entry in the home page, to host both CSW and STAC (both catalogs, but with very different
targets).
Per service collections vs everything collections
This is another fundamental how should it work question. If you look at our services each has been done independently with its own collects list. The other implementations (for example pygeopai) have a single collections list; which has more functionality dumped into it to reflect each ogcapi-service conformance enabled …
The “everything” collections is a much better user experience. List stuff and how to interact with it in one place. However it has its limits; even pygeoapi has ended up with “collection” and “record-collection” now…
Keeping them independent results in a service that is small enough we could test openapi client generation (but perhaps only the two of us share that dream eh?).
It also allows to sustain micro-service efforts like GeoServer cloud. If everything was added at once, we’d have to take some
extra steps to also allow publishing the various services independently. I’m imagining a “api-core” module adding the notion
of “collection” for anything that is data like, and others that add extra bits to complete their functionality: resources, links,
unfortunately also data structures, which is not going to work well in java beans (Document subclasses) that are going to
be encoded in JSON.
In that sense, having a single monolithic implementation makes things easier, one knows in advance which APIs are there,
and what information is needed.
Aside: It looks like some things like ogcapi-records search are designed to be additions to all collections end-points (adding text search for example).
That’s CSW in the API world. You can still search for metadata, althought it adds the ability to search for collectionsn too.
If we could answer this a bit better we could determine if a refactor / merge is needed?
Suggesting a refactor obvisly makes a graduation of the OGC API modules harder, raising the bill for getting that done.
Tech preview
Last year I relaxed Docker restriction to allow for nightly builds and public testing of OGCAPI services. Even putting that example in the user guide for cut and paste ease. I am not sure if it has resulted in any additional feedback/development/funding?
Not that I know of. The feedback I get normally is that OGC APIs are great from people that want “change”, and
cold reactions from the ones that want to get things done, and have them done already with classic OGC services
(why rewrite to get the same functionality, and often, less functionality?).
The obvious answer to me is to deploy them both side by side, and wait for new clients, written from scratch
to use OGC APIs, to become popular and eventually make OWS obsolete.
In many large organizations, that might easily take 10-15 years to complete, but I would not worry about that,
what’s important today is to get that ball rolling.
My other idea is to put up a “tech preview” of GeoServer only with OGCAPI services with a warning that “contains community components that have not been reviewed”. Similar to how Eclipse Foundation includes a warning on downloads that include incubating components…
With only OGC API services… I like the idea, in practical terms that’s hard (these comments are making me wonder how much
you actually grepped through the code). Right now OGC API X is based on the correspoding X-service, building the request object
programmatically and calling the XService Java object (the “engine” for that service).
To go where you want, and limiting attention to Features, we need to:
And then basically rinse and repeat for the other services (for extra fun, consider that today you cannot have WMS without WFS around,
due to GetFeatureInfo depending on GML outputs).
Mind, the above refactors are a good idea in general, not just for OGC API, and while most of it will just be moving classes
around, there is also a part that involves
Goal: Get more eyes on the OGCAPI protocols and thus more interest both development and funding …
Caution: This may not be worth it give our poor community response to participation. I have been disappointed with the response thus the 2024 roadmap challenges. It may be easier to interest people in new functionality; but we do need to look to sustainability also.
The lack of interest seems … common. The FOSS4G-EU OGC API session was not exactly packed (might have been because
it was the last day, but the same room was full for the closing ceremony). On the other hand, I’ve seen well attended sessions
with software that happens to implement OGC API, but markets itself as a lean project, that also happens to implement OGC APIs
Cheers
Andrea
On Sat, Jul 6, 2024 at 9:27 AM Andrea Aime <andrea.aime@anonymised.com…> wrote:
And then basically rinse and repeat for the other services (for extra fun, consider that today you cannot have WMS without WFS around,
due to GetFeatureInfo depending on GML outputs).
Mind, the above refactors are a good idea in general, not just for OGC API, and while most of it will just be moving classes
around, there is also a part that involves
Cliffhanger! Sorry, on the train and got distracted.
There is also a part that involves actual code development and likely some nasty surprises.
Cheers
Andrea
Regarding OGC web services vs OGC API, during the OGC presentation, the speaker mentioned that when they think that a given API is good enough, they will deprecate the matching OGC Web Service.
Alexandre
Le sam. 6 juil. 2024 à 09:29, Andrea Aime <andrea.aime@anonymised.com> a écrit :
On Sat, Jul 6, 2024 at 9:27 AM Andrea Aime <andrea.aime@anonymised.com> wrote:
And then basically rinse and repeat for the other services (for extra fun, consider that today you cannot have WMS without WFS around,
due to GetFeatureInfo depending on GML outputs).
Mind, the above refactors are a good idea in general, not just for OGC API, and while most of it will just be moving classes
around, there is also a part that involvesCliffhanger! Sorry, on the train and got distracted.
There is also a part that involves actual code development and likely some nasty surprises.Cheers
Andrea
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel
Alexandre,
indeed they did. At the same time, the existing OWS APIs have been around for such a long time,
what is the meaning of “deprecating”, besides a formal statement? That will have some effect on
large, formal organisations, but the rest of the world will carry on unscathed.
The worst they can do is to remove the schemas from https://schemas.opengis.net/.… that may
have some effect, but the net result will be to mostly piss off people and give them a reason
to choose an “open specification” rather than an official standard, and just prompt developers
to make sure their software is working offline (AFAIk Geoserver does not need the schemas
over there, effort was put in to work mostly offline, but I may be wrong and/or some
cases might not actually work).
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
Hi All,
I apologize for the slow response and thank you all for your detailed and valuable contributions to this thread.
I am still waiting for funding confirmation, but I want to emphasize the importance of approaching this as a community-driven, collaborative effort.
Once funding is secured, our focus will be on ensuring that the OGC Features API meets production readiness as per customer requirements. Consequently, any refactoring, design, or architectural decisions must be made collaboratively to ensure that other API implementations can seamlessly follow suit where applicable.
To this end, I propose we develop a plan to initially refactor the gs-ogcapi-core and gs-ogcapi-features modules as extensions. Based on the concerns and ideas discussed in this thread, we should work towards the following objectives:
Currently, the service metadata and configuration for the OGC Features module use the WFS service configuration. We need to decouple it from the WFS configuration, and the exact approach is open for discussion.
The OGC Features API is divided into separate specifications, some of which are not finalized. It is crucial that these unfinished aspects of the API can be enabled or disabled via configuration. This configuration will likely be part of the ogcapi-specific service configuration mentioned above.
We need to establish and integrate continuous integration and deployment pipelines for automated testing. These pipelines should run CITE conformance tests and additional integration tests (e.g., against PostGIS).
A technical preview distribution of GeoServer with the OGC APIs enabled will allow us to reach a wider audience for testing and early feedback.
To complement community discussions on design and implementation, it would be ideal to organize an online code sprint. Although we may not have sufficient funding for an on-site event, an online code sprint would still enable all interested parties to contribute focused time and effort.
Thank you again for your initial thoughts. I look forward to getting started and hearing from all of you.
Best regards,
Gabe
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
Thanks Gabe,
I am going to tone down some of the requirements as I am still learning what is going on.
I am not sure if the WFS configuration needs to be completely independent. We have global WFS settings that are used for workspace WFS (even if the global service is turned off). So I do not think it would be terrible if WFS Settings were used to control how Features are served.
It would be more important to preserve data directory compatibility than anything.
There may be other ways to approach, the user interface could always change to say “Features” even if the configuration file remains wfs.xml for example.
For the enable/disable “conformance classes” - my customers wish this level of control, even if a feature is finalized they may not wish to use it.
So it is my hope that we can figure out how to enable/disable. And then perhaps then the specifications that are not finalized can have a sensible default of “disabled”.
Jody Garnett
camptocamp
INNOVATIVE SOLUTIONS
BY OPEN SOURCE EXPERTS
Gabriel Roldán
Geospatial Developer
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
Hi Gabriel,
thanks for following up, some feedback inline below.
- Defining and Implementing Service Metadata Independent of WFS Configuration:
Currently, the service metadata and configuration for the OGC Features module use the WFS service configuration. We need to decouple it from the WFS configuration, and the exact approach is open for discussion.
Agree with Jody, I don’t see it as a strict necessity to separate the two. Even when separate, there is a number of output format configurations in the WFS admin
page, which are stored in the WFSInfo, that would still be shared between the two services.
Unless one wants to configure the output formats in different ways depending on the service? Seems unlikely but not impossible.
Generally speaking, in my mind, I see parts that are specific to WFS/Features (e.g., titles, descriptions, keywords), parts that are shared (output formats)… but going there
would mean breaking the existing configuration and REST API quite a bit.
I’d wait and see how the requirements from actual usage emerge.
- Designing and Implementing a Mechanism to Enable/Disable Non-Finalized Parts of the Specification:
The OGC Features API is divided into separate specifications, some of which are not finalized. It is crucial that these unfinished aspects of the API can be enabled or disabled via configuration. This configuration will likely be part of the ogcapi-specific service configuration mentioned above.
This is going to be “fun”. It could be something similar to KVPReader, pluggable objects that take the raw request and can, if enabled, enrich/alter the
request object which is being sent to the underlying feature engine.
- Setting Up CI/CD Pipelines:
We need to establish and integrate continuous integration and deployment pipelines for automated testing. These pipelines should run CITE conformance tests and additional integration tests (e.g., against PostGIS).
I would recommend against setting up a CI/CD in Jenkins, if that breaks, we are too late as the commit already got in,
and grabbing the original committer might take time. Instead, make it a Github action that fails in the PR, blocking its merger.
Testing can be “fun”, the CITE tests are not data specific, meaning that you can have different results depending on which
data you use for the tests. I’d start with the WMS 1.0 CITE test data directory, as it has simple data, while the WFS ones can be
more challenging. And you could have the GeoServer under CITE use PostGIS, to catch two birds with one stone
(again, some failures will be format specific, a test might work under PostGIS and fail under GeoPackage for example).
- Providing a Technical Preview Distribution of GeoServer:
A technical preview distribution of GeoServer with the OGC APIs enabled will allow us to reach a wider audience for testing and early feedback.
Isn’t testing with the docker image enough? Even if they reach extension status, they would still be optional, but easy to install.
- Organizing a Code Sprint:
To complement community discussions on design and implementation, it would be ideal to organize an online code sprint. Although we may not have sufficient funding for an on-site event, an online code sprint would still enable all interested parties to contribute focused time and effort.
Even an online sprint will require some funding, a large part of the participation cost is normally lost revenue.
Cheers
Andrea