Install COG Plugin on Docker

Hi all,

I’m trying to install the COG plugin on my GeoServer (2.26.0) that is running on a docker, but I’m not having any success.

I followed the docker installation page, and used “cog” in the community_extensions parameter, but it didn’t work. I tried putting “cog-plugin”, that didn’t work either.

https://docs.geoserver.org/main/en/user/installation/docker.html

I then went to the Community Modules plugins page, and then tried “cog-s3-plugin”, “cog-google-plugin” and nothing worked

https://build.geoserver.org/geoserver/2.26.x/community-latest/

Could anyone give me a tip, so I can solve this problem and install the plugin?

Best Regards,

Fernando Quadro
https://fernandoquadro.com.br

Hi Fernando

Does the kartoza docker image allow you to download and install extensions from a local folder? I know GitHub - geoserver/docker: GeoServer docker image does.

You can download the community extensions that are built nightly from the link you have already seen: Index of /geoserver/2.26.x/community-latest/

As per COG (Cloud Optimized GeoTIFF) Support — GeoServer 2.27.x User Manual, you would need to select one of these plugins only, e.g. cog-s3 (I see you just specified cog or cog-s3-plugin)

I hope this helps you get started

Peter

Hi Peter,

Thanks for the reply, and it does allow me to download the plugins, so much so that I installed S3 Geotiff and it downloaded normally, only the COG is not downloading and is giving the following error in the log:

25 Nov 17:39:42 WARN [org.geoserver] - Could not load map entry, skipping as it might have put by a plugin that’s no more available
com.thoughtworks.xstream.mapper.CannotResolveClassException: cogSettings
at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)

It seems to me that to install the plugin it is not “cog”, “cog-s3”, and no other that I have tried.

Can anyone who has already installed it in docker tell me how to do it?

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

The cog plugin was split up recently due to conflicting libraries between the different cloud services. I am not sure there is one just called cog anymore - checking the nightly build folder and it is true. No community module named cog.

But really these are shared to get feedback and funding. So if you are interested in this functionality please contact the developer to ask how to help.

Jody,

How can I talk to the developer? What would be the email address so I can contact him and ask this question, please?

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

Uh, “the developer” is actually a group of people that contributed each a bit. The people most knowledgeable about the module are from Geosolutions right now.

That said, I’d suggest to split the issue first: try using a nightly build and see if it works (all bits should be nightly, don’t mix release and nightly). Does that work? If so, talk to the people responsible for the docker image. Otherwise report your findings here

Cheers
Andrea

Il lun 25 nov 2024, 19:50 Fernando Quadro via OSGeo Discourse <noreply@discourse.osgeo.org> ha scritto:

fernandoquadro
November 25

Jody,

How can I talk to the developer? What would be the email address so I can contact him and ask this question, please?

Best regards,

Fernando Quadro
https://fernandoquadro.com.br


Visit Topic or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, click here.

Andrea,

I’ll test it with the nightly version, but if it works, won’t I be able to use it on a stable version with just the “COMMUNITY_EXTENSIONS” directive in the Docker configuration?

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

Andrea,

I downloaded GeoServer 2.26-SNAPSHOT locally on my machine, and followed the same procedure as in production, installing the 2 plugins: first, only COG-S3, but it does not appear in the “Store” menu, then installing S3 Geotiff, and this, as it had already happened in production, worked.

Unfortunately, GeoServer does not complain about anything in the log, it is as if the plugin had not been installed.

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

The COG extension does not add a new store, but options in the existing Geotiff store

Cheers
Andrea

Il lun 25 nov 2024, 21:21 Fernando Quadro via OSGeo Discourse <noreply@discourse.osgeo.org> ha scritto:

fernandoquadro
November 25

Andrea,

I downloaded GeoServer 2.26-SNAPSHOT locally on my machine, and followed the same procedure as in production, installing the 2 plugins: first, only COG-S3, but it does not appear in the “Store” menu, then installing S3 Geotiff, and this, as it had already happened in production, worked.

Unfortunately, GeoServer does not complain about anything in the log, it is as if the plugin had not been installed.

Best regards,

Fernando Quadro
https://fernandoquadro.com.br


Visit Topic or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, click here.

Andrea,

It worked, both in the nightly version and in the stable version, but on Windows (my machine).

Could you ask the GeoSolutions team what name I need to declare in COMMUNITY_EXTENSIONS in Docker (I’m using the Kartoza image) so that it enables the plugin, because all the references I tried, GeoServer complains that it can’t load. See:

25 Nov 17:39:42 WARN [org.geoserver] - Could not load map entry, skipping as it might have put by a plugin that’s no more available
com.thoughtworks.xstream.mapper.CannotResolveClassException: cogSettings

I base myself on this documentation:

https://docs.geoserver.org/main/en/user/installation/docker.html
https://docs.geoserver.org/latest/en/user/community/cog/cog.html

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

Geosolutions does not contribute to, nor use, the Kartoza docker images. You should ask in the Github repository for that image

Cheers
Andrea

Il lun 25 nov 2024, 22:56 Fernando Quadro via OSGeo Discourse <noreply@discourse.osgeo.org> ha scritto:

fernandoquadro
November 25

Andrea,

It worked, both in the nightly version and in the stable version, but on Windows (my machine).

Could you ask the GeoSolutions team what name I need to declare in COMMUNITY_EXTENSIONS in Docker (I’m using the Kartoza image) so that it enables the plugin, because all the references I tried, GeoServer complains that it can’t load. See:

25 Nov 17:39:42 WARN [org.geoserver] - Could not load map entry, skipping as it might have put by a plugin that’s no more available
com.thoughtworks.xstream.mapper.CannotResolveClassException: cogSettings

I base myself on this documentation:

https://docs.geoserver.org/main/en/user/installation/docker.html
https://docs.geoserver.org/latest/en/user/community/cog/cog.html

Best regards,

Fernando Quadro
https://fernandoquadro.com.br


Visit Topic or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, click here.

Andrea,

And what about the OSGeo images (docker.osgeo.org), do you contribute those? I have no problem changing the “source” of the kartoza image to osgeo.

If so, can you tell me what name I need to pass in the “COMMUNITY_EXTENSIONS” parameter?

I appreciate your help.

Best regards,

Fernando Quadro
https://fernandoquadro.com.br

It is a bit disappointing that the kartoza docker image is making use of our build in the fashion shown in your screen snap.

I occasionally help with the geoserver docker image, and added the ability to try community plugins with the nightly builds in order to get more feedback / funding. This functionality downloads from the nightly build server where both extensions, select community modules, and the war are published.

The release images use source forge download location. Since the release process does not include community modules so you cannot mix them into a release such as 2.26.1 as you are trying to do in your screen snap.

You may wish to check out the following discussion: GEOS-11555: fix installation of extensions in various configurations by spike83 · Pull Request #102 · geoserver/docker · GitHub which contains a similar discussion and the suggestion to make a change proposal.

Notes:

  • If people wish to allow community modules to be used against release docker it needs to be clear that it is an experiment only. The community modules are build each day and the specific libraries and so on used change over time. So an image that works one day, may not work the next time you start it up.

  • Developers can download the source code for 2.26.1 and build the community module. This provides something that at least compiles with the same set of libraries used in the release. This is what I do for my commercial customers when when they ask to preview a community module that they are paying me to work on.

On Mon, Nov 25, 2024 at 11:07 PM Fernando Quadro via OSGeo Discourse <noreply@discourse.osgeo.org> wrote:

fernandoquadro
November 25

Andrea,

And what about the OSGeo images (docker.osgeo.org), do you contribute those? I have no problem changing the “source” of the kartoza image to osgeo.

“You” as in me, no, I take no part in docker image development, although I occasionally enjoy using an official
GeoServer image (OSGeo one) to quickly have a specific version/extensions ready to test without having to assemble
the war manually.

“You” as in GeoSolutions, again, no, the company has its own docker image, that’s open source, but without a community
around, and is taking a different approach compared to Kartoza/GeoServer own images.
In particular, the extensions/community modules are baked into the image at build time, my colleagues working on it
make a specific build for each customer and deployment. The downside is evident, lots of docker image building,
the upside is that once built it is stable, no risk of deploy failures due to networking issues, and instant deploy,
no need to wait for extensions/community modules to be downloaded.

Cheers
Andrea

On Tue, Nov 26, 2024 at 6:25 AM Jody Garnett via OSGeo Discourse <noreply@discourse.osgeo.org> wrote:

  • If people wish to allow community modules to be used against release docker it needs to be clear that it is an experiment only. The community modules are build each day and the specific libraries and so on used change over time. So an image that works one day, may not work the next time you start it up.

The non-repeatable deploy is indeed a problem… but I guess it can be eliminated by downloading the community module separately,
and then mounting it like suggested here? (mind, I did not try it out):
https://github.com/geoserver/docker?tab=readme-ov-file#how-to-install-additional-extensions-from-local-folder

Should work fine for at least simple docker usage, not sure what would happen in a more formal environment
like Kubernetes (maybe it can be made to work by mounting a network share, not sure).

Cheers
Andrea

Andrea and Jody,

Thanks for the clarification. From what I understand, it shouldn’t be allowed to use the COMMUNITY_EXTENSIONS parameter in stable versions (via docker). Is that it?

I’ll take a look at the link Andrea suggested (GitHub - geoserver/docker: GeoServer docker image)

Best Regards,

Fernando Quadro
https://fernandoquadro.com.br

Andrea is correct that it is indeed a risk, rather than not allowed.

Andrea I am more against having mixed message; I suggest a proposal to obtain permission to distribute the community module. They can have a warning about being unstable etc…

Looking at checklist:

  • module is not site-specific
  • designated and active maintainer (not required)
  • considered “stable” by the majority of the PSC (not required)
  • 40% test coverage (run tests for stability, but do not require coverage target)
  • no IP violations (not required)
  • page in the user manual
  • maintainer has signed the GeoServer Contributor Agreement (required)

I think that reduced check list would be just fine to share community module?

Jody, I think you have a tendency to make things more complex than necessary…

Community modules are a well established concept in GeoServer, citing from the documentation:

Warning
Community modules are generally considered experimental in nature and are often under constant development. For that reason documentation in this section should not be considered solid or final and will be subject to change.

Note
Unlike the official extensions, these modules are not released and stored on SourceForge when an official GeoServer release is produced. As a consequence, using a module built against the nightly build of GeoServer with an official release won’t probably work.

If you need a community module for an official release, the only way is to build it with the sources of the GitHub tag matching the release.

Also, community modules are already being distributed, otherwise nobody would give a damn about them. We get funding to push them forward because people can actually use them, otherwise nobody would care.

Now, docker provides a different way to download things and people might just be ignoring the wealth of warnings and docs we have built around community modules to warn people… docker users need to be communicated in a way they understand, and that would make the realize what they are doing… the faint of heart please stop reading here because I’m going to say it: “non repeatable deploy”!.

The way the official GeoServer docker image is setup means playing russian roulette with the deploy the moment a community module steps in:

  • If a nightly build of both GeoServer and the community module are deployed toghether, they are at least from the same code base, but one is exposed to all the changes happening between one “docker run” and the next. For extra fun, if it’s a cluster, it means running different code in different nodes. Also for reference, GeoServer receives around 2 new commits per day.
  • If a release is deployed with a community module, then the core is stable, but the community module is new at every deploy, with a distance between the two that grows over time.

On top of that, nightly builds of older series get eventually removed, so it is guaranteed that after a few years that one deployment will be bricked.

Just have them read this when they look for a way to deploy community modules, and then let them decide what to do. Hopefully that will make someone bend towards shelling out some funding, or developer time, to get the module they need in extension. It has to be in this order, “I verified I need it, it’s part of my deployed infrastructure and it was working, it’s then worth spending money on, rather than spending redoing the infrastructure with a different product”. Make them choose between two ways to spend money, that they have to spend anyways.
Then funding finally shows up, rather than just telling developers they should be doing this or that on a Sunday, for free, for some stranger in urgent need, instead of spending time with their family or getting some rest.

We don’t need new rules for semi-supported community modules, we just need the have the user base aware of what they are getting themselves into.

Cheers
Andrea

I was looking towards getting permission to publish select community modules as part of the release.

Sticking with status-quo and adding “non repeatable warning” in the docker logs is a nice idea!

I was looking towards getting permission to publish select community modules as part of the release.

On no… modules that are listed as community are known to have some sort of problem, we don’t want to make them easy to get.
Let’s take the subject of this discussion, COG modules… what does it need to be distributable?
The following:

  • Debug and fix usage of internal binary masks, they are been reported not working, we don’t know why
  • Rewrite the way the tile directory is read, it’s now fully loaded in memory with a small request buffer, makes very large COGs (100+GB) unusably slow. (and yes, there is a system variable to tune the buffer size, as a workaround)
  • For each cloud, build an online test that runs and verified the module is working. This should be done either using an emulator (e.g., Azurite, Minio) or even better, have someone cough up the money to run multiple builds a day against the actual cloud itself. That makes for 3 cloud specific online tests, plus the generic HTTP one that could be simply coded against Wiremock.

The first two items need to be done before we consider moving the module to extension. The third part could be done per single online test, e.g., we could have maybe just generic HTTP supported at the beginning, and wait for specific cloud funding to show up so that we can build the necessary online tests

Cheers
Andrea