[SAC] [OSGeo] #2438: Create a geos and postgis docker repo on repo.osgeo.org

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+-----------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Keywords:
---------------------------+-----------------------
This is mostly to replace the private registry strk is currently running.

We will start by using it to hold images for our geos/postgis bots and
maybe eventually expand the use for other things.

These should be separate repositories/

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

For permissions add robe, strk for starters. We'll add one for jenkins
bot later.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:1&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by jive):

I setup the postgis one for your review:

* postgis-docker repository created, allowed anonymous access, but turned
off v1 api access
* docker group now includes postgis-docker
* setup postgis-admin role giving it permissions for the postgis-docker
repository
* robe and strk users have been granted postgis-admin role (they were both
admins already but whatever)
* created a local user postgisbuild user with the above postgis-admin
role, it uses robe's email for notifications (please adjust this user as
needed for your jenkin). This mirrors what was done for geoserver jenkins.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:2&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by jive):

Please try out the above repository, and write up how it works for the
wiki page!

General approach:

1. Setting up your osgeo userid credentials in ~/.docker/config.json
2. Push to https://repo.osgeo.org/repository/postgis-docker/
3. Please pull anonymously from https://repo.osgeo.org/repository/docker/

References:

* https://blog.sonatype.com/using-nexus-3-as-your-repository-part-3
-docker-images
* https://help.sonatype.com/repomanager3/formats/docker-registry/pulling-
images
* https://help.sonatype.com/repomanager3/formats/docker-registry/pushing-
images
* https://help.sonatype.com/repomanager3/formats/docker-
registry/authentication

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:3&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by strk):

Instructions
https://blog.sonatype.com/using-nexus-3-as-your-repository-part-3-docker-
images

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:4&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by strk):

Do we need a specific port ?

{{{
[strk@liz:~] docker login repo.osgeo.org
Username: strk
Password:
Error response from daemon: login attempt to https://repo.osgeo.org/v2/
failed with status: 404 Not Found
}}}

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:5&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

strk that's what I was saying that's the same error I get.

All the videos I've seen they explicitly have docker registry run on a
separate port. Because you can't give a path for login.

I think we could just setup another domain like docker.osgeo.org and have
it connect to port 8083 or something in nexus. But wasn't sure if there
was a way around that.

So I'm thinking we set docker to explicitly be on specific port (of course
I would need to expose that too on the nexus docker (or maybe not maybe
that could be a path)

and then docker.osgeo.org goes to that.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:6&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

aha here it describes the issue

https://help.sonatype.com/repomanager3/formats/docker-registry/ssl-and-
repository-connector-configuration

{{{
The docker client does not allow a context as part of the path to a
registry, as the namespace and image name are embedded in the URLs it
uses. This is why requests to repositories on the repository manager are
served on a specific and separate port from the rest of the application
instead of how most other repositories serve content via a path i.e.
<nexus-hostname>/<repositoryName>/<path to content> .
}}}

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:7&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

This one is interesting

https://blog.sonatype.com/setting-up-a-docker-private-registry-with-
authentication-using-nexus-and-nginx

It uses a single nginx proxy config, but if the agent is docker, then
redirects to registry port, but don't see a way of getting around need to
open an additional port on nexus docker container and if we have more than
one docker reigstry I think we'll need a port for each so we should just
put them in now.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:8&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by strk):

DISCLAIMER: I did not read the articles in those links

Are you saying that Nexus does not allow having different
permissions for writing in different subdirs ?

I do like the idea of using docker.osgeo.org

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:9&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

strk,

I don't understand the link of registry vs. folder etc.

My understanding is with registry which is the way I think jive has it set
up -- each registry can completely manage their roles/etc. I think if
it's a single registry you can't but maybe one registry is sufficient.

But anyway the whole path issue is more of a limitation in docker itself,
not nexus. That docker registries need to authenticate at the root
because the path is encoded in the tag. DISCLAIMER - I may not know what
I am talking about.

At anyrate thinking of copying over nexus container on osgeo4 to
experiment. Need a backup there anyway. I think even though in theory I
can open up ports on a docker container running -- it's not supported, so
I'd rather shut it down and start it up with many ports which means there
would be like 5 minutes of downtime while we do this. If there is no way
around this whole having to run in a port.

I would think with nginx -- we could point a path like docker.osgeo.org ->
nexus.lxd:8081/docker

but I have not seen anyone doing that so maybe it's not doable.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:10&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

Okay I was able to successfully login on my dev container. Had to add
Docker Bearer Token Realm (to the nexus -> Realm ) section, in addition to
using a separate port.

I'm still unclear if we can get away with just a single port for all
docker repositories. Still experimenting with that.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:11&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

found this thread which seems to do it without additional ports, just
rewriting the docker calls

https://stackoverflow.com/questions/47178055/nexus3-push-to-docker-group-
repo

I'll give that a try in dev.

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:12&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

okay tried in dev and worked, I put in place on repo.osgeo.org (without
specifying any ports), just rewriting the path calls and was then able to
log in with

{{{
docker login
}}}

https://git.osgeo.org/gitea/sac/osgeo3/commit/c48afd1b84a1c1c85a831cfa6a51f291311d6f1d

But I haven't tried committing (and not sure what paths should be put in
for push to differentiate the repos.

I was able to push in dev (but that was with port explicitly for postgis-
docker, and then it appeared when I browsed both the docker and postgis-
docker)

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:13&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+------------------------

Comment (by robe):

Okay I think I got this working. I created a new docker.osgeo.org nginx
config dedicated. I could put it all in the nexus one, but felt might be
better to keep it separate. I'm planning to eventually take out all the
/v2 stuff I put in on the nexus config.

So the way it works, all pushes must go thru the project repo

e.g postgis-docker.osgeo.org, geoserver-docker.osgeo.org, geos-
docker.osgeo.org

But pulls go thru

docker.osgeo.org

To test I copied over strk's images:

{{{
docker pull docker.kbt.io/postgis/build-test:trisquel2
docker tag docker.kbt.io/postgis/build-test:trisquel2 postgis-
docker.osgeo.org/postgis/build-test:trisquel2
docker push postgis-docker.osgeo.org/postgis/build-test:trisquel2

}}}

and that shows a new image in postgis-docker repository folder (and of
course exposed in the docker group

committed at -
https://git.osgeo.org/gitea/sac/osgeo3/commit/87932245f05841f0413053e5f824dc0cd5bfae46

So key area of nginx script looks like this sorry about the crappy
indentation, going to fix that next

{{{
  location ~ ^/(v1|v2)/[^/]+/?[^/]+/blobs/ {
            if ($request_method ~* (GET) ){
                         rewrite ^/(.*)$ /repository/docker/$1 last;
                 }
            if ($host = postgis-docker.osgeo.org ){
                 rewrite ^/(.*)$ /repository/postgis-docker/$1 last;
             }
           if ($host = geos-docker.osgeo.org ){
                 rewrite ^/(.*)$ /repository/geos-docker/$1 last;
           }
           if ($host = geoserver-docker.osgeo.org ) {
                 rewrite ^(.*)$ /repository/geoserver-docker/$1 last;
           }

             rewrite ^/(.*)$ /repository/docker/$1 last;
         }

       location ~ ^/(v1|v2)/ {
                 if ($request_method ~* (GET) ){
                         rewrite ^/(.*)$ /repository/docker/$1 last;
                 }
                 if ($host = postgis-docker.osgeo.org ) {
                         rewrite ^/(.*)$ /repository/postgis-docker/$1
last;
                 }
                 if ($host = geos-docker.osgeo.org ) {
                         rewrite ^/(.*)$ /repository/geos-docker/$1 last;
                 }
                 if ($host = geoserver-docker.osgeo.org ) {
                         rewrite ^/(.*)$ /repository/geoserver-docker/$1
last;
                 }

                 rewrite ^/(.*)$ /repository/docker/$1 last;
         }

     location / {
                 # First attempt to serve request as file, then
                 # as directory, then fall back to displaying a 404.
                 #try_files $uri $uri/ =404;
                 client_max_body_size 0;
                 include /etc/nginx/proxy_protocol_params;
                 #need to change this if using https on server and have a
redirect
                 proxy_pass http://nexus.lxd:8081;
                 proxy_redirect off;
         }
}}}

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:14&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+------------------------
Changes (by robe):

* status: new => closed
* resolution: => fixed

Comment:

I'm going to close this out. I tested on one of the dronie agents by
doing this

{{{
docker rmi docker.kbt.io/postgis/build-test:trisquel2 #if I don't do this
then it just tags the pulled (smart enough to know it's already been
pulled from another server)

docker pull docker.osgeo.org/postgis/build-test:trisquel2

}}}

I also setup geos-docker and pushed the docker.kbt.io/geos/build-
test:alpine

jive - when you get the chance, can you confirm your group can push to

{{{
docker login geoserver-docker.osgeo.org
docker push ...
}}}

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:15&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.

#2438: Create a geos and postgis docker repo on repo.osgeo.org
---------------------------+------------------------
Reporter: robe | Owner: sac@…
     Type: task | Status: closed
Priority: normal | Milestone: Unplanned
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+------------------------

Comment (by jive):

We are just trying it out now (to store some cite testing images).

Q: Do you know how the docker hub https://hub.docker.com/u/osgeo is
managed?

--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2438#comment:16&gt;
OSGeo <https://osgeo.org/&gt;
OSGeo committee and general foundation issue tracker.