RE: Physical movement of machines preparation

Committees SAC SAC mailing list
, ,
1

We are only migrating: osgeo6, osgeo7, osgeo8 and osgeo9
but osgeo6 is going to be reformatted so we can use it for virtualization.
During the physical movement services on the moved machine will be
shutdown.

My proposed way of doing the migration:
- Migrate osgeo9
  - this would keep alive tracsv
  - when turning back on the services will continue running normally
- Do the final steps, mentioned above, for using the containerized mail
server on osgeo9
- Migrate osgeo7
- Migrate osgeo8
- Migration of osgeo6:
  - A new disk needs to be bought and installed
  - Lance will do us the favor of installing and formatting and initial
installation of operative system for virtualization
  - He can do on the current datacenter or the new datacenter
- Migrate contents of osgeo3 and osgeo4
  - staging can go directly to osgeo6
  - The rest we can decide where to put the conainers
- Shutdown osgeo3 and osgeo 4

Please let me know your comments

Regards
Vicky

Regardless what order we move the other servers, I think osgeo6 physical server should be moved first.

From what Lance mentioned, it sounded like we'd have at most a couple of hours downtime.
Whatever services we can't handle that much downtime, should be moved to osgeo6 at least temporarily once osgeo6 server is at its new home.

If we agree no service will suffer from more than 3 or 4 hours downtime, then I would say

2) Copy over all containers on osgeo9 to osgeo6 just before the move of osgeo9 (note at this point osgeo6 server is already at its new physical location)

That way if anything goes wrong during the transport of osgeo9, we can hopefully restart them all on osgeo6 already moved.
We might want to have secure though running in osgeo6 for the duration of the move since that is the only service I can think of where it's really bad to have that much downtime, since qgis plugins and sshing into any server is impacted by that.

3) I really would like to reformat osgeo7 since it's gone thru like 3 ubuntu upgrades and needs to go thru 2 more and it has the oldest OS of all the hosts and was our first learning of lxd so I feel things aren't
setup in the best way.

If we don't do a reformat, then we should perhaps temporarily/permanently move tracsvn to osgeo9

The other items on osgeo7 besides tracsvn /download/photoprism I feel should be moved to osgeo8 - so that would mean moving mapserver, pycsw, live to osgeo8

I've been thinking of osgeo8 like a project host, one that contains project specific vms/containers like the grasswiki, grass (already on there), the woodie-server (used mostly by sac and postgis), postgis-demo server, live vms etc.

Download is already replicated on osgeo8, so we could just repoint download.osgeo.org to osgeo8 during the osgeo7 (reformat/upgrade) and repoint back after. During the migration we wouldn't allow uploads to upload.osgeo.org.

So osgeo7 during the move would have nothing on it.
Osgeo7 after the move we would move download back on to it. We would move some other items that are disk heavy to osgeo7 e.g. peertube, photoprism
And everything on osgeo3 would be moved to osgeo7

And then we get rid of osgeo3. Osgeo6 becomes the new backup/staging server after all is said and done.

We also at this point after should start discussing use of Hetzner as in the board meeting, we did discuss at least for redundancy having a dedicated host at Hetzner. The pricing seemed reasonable something like $100-$250 per month for something somewhat comparable to osgeo7.

I lost the link that was provided at the board meeting for pricing, but was very reasonable. I was mostly concerned about the bandwidth cost as that tends to be what eats up your money.
Tom or Vicky you remember that link discussed in board meeting?

2

On Sun, Sep 14, 2025 at 07:56:06PM -0400, Regina Obe wrote:

Regardless what order we move the other servers, I think osgeo6 physical server should be moved first.
From what Lance mentioned, it sounded like we'd have at most a couple of hours downtime.

This should be done AFTER osgeo6 stops hosting any service,
so AFTER its containerized version takes its place, right ?

I think that's the very first step to complete, probably worth its own
thread to not mix it with the planning of physical movements.

2) Copy over all containers on osgeo9 to osgeo6 just before the move of osgeo9 (note at this point osgeo6 server is already at its new physical location)

Moving containers from a machine to another implies a change in the
public IP address of the service, correct ? So we should also factor
in the downtime due to DNS propagation. We should plan for it
beforetime to have a quick way to update them (lowering TTL) - doing
DNS via Ansible should help - see Making sure you're not a bot!

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

3

On Mon, Sep 15, 2025 at 10:47:36AM -0400, Regina Obe wrote:

1) Containerize current osgeo6 (which is already done on osgeo9)

I understand this step is not completed until the IP (or hostnames?)
that currently points to osgeo6 start pointing to the new
containerized machine.

I actually wonder what "containerize" really meant, other than having
the ability to have a container that can be fully accessed via its own
IP address (currently done with network forward which is one of the
possible options, is my understanding).

2) Confirm all is working on osgeo9 (we did not test the MTA too much but we did test mailing lists), and sending mail via new osgeo6

We'll need other testing ONCE the IP (or hostnames) are repointed.

3) Production Switch to containerized osgeo6 on osgeo9

Is this the point in which you would assign 140.211.15.3 to osgeo9 ?
Or do you think we should ONLY point the hostnames mail.osgeo.org and
lists.osgeo.org and tilecache.org and www.tilecache.org ?)

4) shut off osgeo6, make sure everything is still working

I'll not think about the next steps until the previous steps
(up to "production switch") are clear.

DNS wise I think our downtime is pretty low. I've noticed it only takes like an hour or so.

The mail.osgeo.org and lists.osgeo.org DNS records have a TTL of
300 seconds (5 minutes), way less than an hour. And we can now change
them easily via Ansible, if we want, see:

  Making sure you're not a bot!
  Making sure you're not a bot!

Though we can't account for dns servers caching for longer than they should or client pcs caching.

If we cannot account for DNS timing, how do we deal with mail queues ?

Should we configure the current mail server to forward ALL incoming mail to the new mail
server, thus keeping 2 separate IP addresses and 2 hostnames, and refuse to work as a mail
submission agent (block all outgoing mail) ?

Or do we keep the SAME IP and thus the same DNS records while just
assigning them to osgeo9 instead of osgeo6 ?

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

4

> Regardless what order we move the other servers, I think osgeo6 physical
server should be moved first.
> From what Lance mentioned, it sounded like we'd have at most a couple of
hours downtime.

This should be done AFTER osgeo6 stops hosting any service, so AFTER its
containerized version takes its place, right ?

Correct. Plan is
1) Containerize current osgeo6 (which is already done on osgeo9)
2) Confirm all is working on osgeo9 (we did not test the MTA too much but we did test mailing lists), and sending mail via new osgeo6
3) Production Switch to containerized osgeo6 on osgeo9
4) shut off osgeo6, make sure everything is still working
5) put in new disks in osgeo6 so it has a lot more space
6) reformat osgeo6 physical and bring back up as an lxd host

I think that's the very first step to complete, probably worth its own thread to
not mix it with the planning of physical movements.

> 2) Copy over all containers on osgeo9 to osgeo6 just before the move
> of osgeo9 (note at this point osgeo6 server is already at its new
> physical location)

Moving containers from a machine to another implies a change in the public IP
address of the service, correct ? So we should also factor in the downtime due
to DNS propagation. We should plan for it beforetime to have a quick way to
update them (lowering TTL) - doing DNS via Ansible should help - see
Making sure you're not a bot!

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

DNS wise I think our downtime is pretty low. I've noticed it only takes like an hour or so.
Though we can't account for dns servers caching for longer than they should or client pcs caching.
But that said, this is a PLAN B - only would be done if osgeo9 for whatever reason doesn't make the physical move
And we are forced to restart all services back on osgeo6.

I'm hoping we wouldn't have to execute on this plan B.

The only case I was thinking was with secure, that we might want to switch it temporarily to osgeo6, but even that I think people can live for a couple of hours
without secure being live. Just means that only SAC would be able to ssh into the machines and that people wouldn't be able to upload plugins to QGIS for that time period
or log into any of our services.

5

On Mon, Sep 15, 2025 at 10:47:36AM -0400, Regina Obe wrote:

> 1) Containerize current osgeo6 (which is already done on osgeo9)

I understand this step is not completed until the IP (or hostnames?) that
currently points to osgeo6 start pointing to the new containerized machine.

We are changing the ip to what is currently in the network forward in osgeo9 but keeping the host name.
I think we'd want to keep the current osgeo6 ip cause it would be hard to reach it except thru the private ip (which requires you be connected to OSUOSL VPN) if we try to switch
and we already allocated an extra ip for osgeo9 for this purpose.

I actually wonder what "containerize" really meant, other than having the
ability to have a container that can be fully accessed via its own IP address
(currently done with network forward which is one of the possible options, is
my understanding).

There are two kinds of things you can have in lxd. Originally lxd when we started only supported containers.
You can have a VM or an OS container. Container is lighter weight than VM cause it doesn't have it's own kernel and settings on it are really quotas than true settings.
e.g. you allocate disk or memory or cpu it’s a quota. So to change it you just change the quota. No need to even restart the container to do that, where as changing VM disk and memory requires changing config, doing the usual resize disk etc and restarting the VM.

VM emulates everything. We used lxd-migrate tool to replicate osgeo6 to a container.

Lxd-migrate is part of lxd source code lxd/lxd-migrate at main · canonical/lxd · GitHub and they provide the binary as part of their release cycle. It is used both for containerization and virtualization of a Physical/VM

Converting a physical to a VM is trickier as you got to have more stuff configured and I think even have the original install media since it tries to emulate everything.
For most cases unless you are trying to emulate a GUI or a non-Linux OS, OS containers are sufficient and much simpler to manage.

To do say FreeBSD, Windows, or Mac you would need a VM.

VM has one big advantage aside from supporting more OS in that you can nest it more.
E.g. for Bas's cowbuilder Debian testing to emulate the 32-bit issues on PostGIS he was running into, I had to create a VM to run cowbuilder in.
Cause I think cowbuilder tries to create another QEMU of 32-bit or some such thing. So in theory you can emulate a 32-bit within a VM on lxd.

VMs in LXD leverages QEMU under the hood but allows you to manage the VM with the same tools as you manage containers.
So they are referred to as instances if you are talking about both VMS and Containers under LXD.
QEMU under LXD has a deficiency or at least it used to in that a 64-bit host can't create a 32-bit VM where as with raw QEMU you can do that.
But the cowbuilder thing, I was able to create a 32-bit VM under a 64-bit lxd vm. Sounds crazy but it works.

> 2) Confirm all is working on osgeo9 (we did not test the MTA too much
> but we did test mailing lists), and sending mail via new osgeo6

We'll need other testing ONCE the IP (or hostnames) are repointed.

> 3) Production Switch to containerized osgeo6 on osgeo9

Is this the point in which you would assign 140.211.15.3 to osgeo9 ?
Or do you think we should ONLY point the hostnames mail.osgeo.org and
lists.osgeo.org and tilecache.org and www.tilecache.org ?)

As mentioned above the plan is to only repoint the host name

> 4) shut off osgeo6, make sure everything is still working

I'll not think about the next steps until the previous steps (up to "production
switch") are clear.

> DNS wise I think our downtime is pretty low. I've noticed it only takes like
an hour or so.

The mail.osgeo.org and lists.osgeo.org DNS records have a TTL of
300 seconds (5 minutes), way less than an hour. And we can now change them
easily via Ansible, if we want, see:

  Making sure you're not a bot!
deployment/src/branch/master/deployment/roles/dns-
records/tasks/mail.yml
  Making sure you're not a bot!
deployment/src/branch/master/deployment/roles/dns-records/tasks/mx.yml

> Though we can't account for dns servers caching for longer than they should
or client pcs caching.

If we cannot account for DNS timing, how do we deal with mail queues ?

We don't. But as I recall I think mail servers are set to retry for like 72 hours by default if they get a bounce.
So I don't see this as being an issue except for making sure osgeo6 queue is clear before we switch over.

Should we configure the current mail server to forward ALL incoming mail to
the new mail server, thus keeping 2 separate IP addresses and 2 hostnames,
and refuse to work as a mail submission agent (block all outgoing mail) ?

That's a good idea. Yes we should probably do that.

Or do we keep the SAME IP and thus the same DNS records while just
assigning them to osgeo9 instead of osgeo6 ?

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

No we do not keep the same ip as mentioned or at least that was not the plan.
As to whether it's better to do that perhaps.

6

I actually wonder what "containerize" really meant, other than having the
ability to have a container that can be fully accessed via its own IP address
(currently done with network forward which is one of the possible options, is
my understanding).

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

Regarding containerizing, that is what I did with the old VMs in osgeo3 and osgeo4 ganetti cluster so I could quickly move them over without having to know how they were set up
e.g. secure.osgeo.osuosl.org, tracsvn.osgeo.osuosl.org, adhoc etc.
At the time the tool was called lxd-p2c (physical to container).

The new tool is called lxd-migrate since the new tool can be used to migrate to a container or a VM.

Details of using it are briefly described here:

https://gitea.osgeo.org/sac/osgeo7/src/branch/master/lxd-migration/README.md

7

On Mon, Sep 15, 2025 at 05:10:39PM -0400, Regina Obe wrote:

We are changing the ip to what is currently in the network forward in
osgeo9 but keeping the host name.

Then, when ready, we should re-point all these A records:

  - lists.osgeo.org
  - mail.osgeo.org
  - (*.)tilecache.osgeo.org [ shall we move this ? ]
  - drone.osgeo.org [ shall we drop this ? ]

And update the SPF records to include the new IP (this could be already done now).

> I actually wonder what "containerize" really meant
>
Lxd-migrate is part of lxd source code lxd/lxd-migrate at main · canonical/lxd · GitHub

What I meant was: are we supposed to have an exact copy of all data ?

I think the answer is YES, but from the time lxd-migrate was run the
data on the machines started diverging. How are we going to deal with that ?

I understood there's an rsync script but that script does NOT copy ALL
the data, just a number of selected subdirectories, and the script itself
is run from within the new machine, making it impossible to copy (say)
the /etc/cron* directories w/out removing the script.

> If we cannot account for DNS timing, how do we deal with mail queues ?

We don't. But as I recall I think mail servers are set to retry for like 72 hours by default if they get a bounce.
So I don't see this as being an issue except for making sure osgeo6 queue is clear before we switch over.

We'll need to make sure mail directed at the old IP get a bounce,
to get the retry (hoping the retry will be to the new IP).

As per OUR queues, at the time of writing mail.osgeo.org has 2114 queued messages.
We need to plan how to clear that queue.

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

8

Then, when ready, we should re-point all these A records:

  - lists.osgeo.org
  - mail.osgeo.org
  - (*.)tilecache.osgeo.org [ shall we move this ? ]
  - drone.osgeo.org [ shall we drop this ? ]

As I recall I think Vicky already moved tilecache to the new osgeo9 mail container, but we probably should move it elsewhere if we want to keep it at all.
It's so old and hasn't been touched in over 10 years, I'm not sure anyone would miss it if it's gone.

Yes drone.osgeo.org should be dropped. That isn't even used.

And update the SPF records to include the new IP (this could be already done
now).

I think we might have done that already. Vicky can you confirm? Or did we do with ansible?

> > I actually wonder what "containerize" really meant
> >
> Lxd-migrate is part of lxd source code
> lxd/lxd-migrate at main · canonical/lxd · GitHub

What I meant was: are we supposed to have an exact copy of all data ?

Not an exact copy of all the data. A lot of the data was archived data (crap in home directories and archives of sites that have long been removed)
So those may exist in the copy we shut down on osgeo7, but for our live, we aren't keeping those folders.

I think Vicky did an analysis of the crons worth keeping and not. Maybe she can speak about that.

We'll need to make sure mail directed at the old IP get a bounce, to get the
retry (hoping the retry will be to the new IP).

As per OUR queues, at the time of writing mail.osgeo.org has 2114 queued
messages.
We need to plan how to clear that queue.

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

Agreed. I suspect those queues if they haven't been sent yet are just garbage and should be flushed.
Probably things like bad email addresses that can't be reached or subscriber spam.

9

On Tue, Sep 16, 2025 at 01:28:15AM -0400, Regina Obe wrote:

As I recall I think Vicky already moved tilecache to the new osgeo9 mail container

She moved tilecache.org to web.osgeo9 where it belongs (good!)
I've now also dropped (*.)tilecache.osgeo.org

Yes drone.osgeo.org should be dropped. That isn't even used.

Also dropped, so we're left with these pointing at osgeo6:

  - lists.osgeo.org
  - mail.osgeo.org

> And update the SPF records to include the new IP (this could be already done
> now).

I think we might have done that already. Vicky can you confirm? Or did we do with ansible?

Not done yet, easy to tell:
https://gitea.osgeo.org/sac/ansible-deployment/src/branch/master/deployment/roles/dns-records/defaults/main/spf.yml

A lot of the data was archived data (crap in home directories and archives of sites that have long been removed)
I think Vicky did an analysis of the crons worth keeping and not. Maybe she can speak about that.

Yes please Vicky, I'm particularly interested in service variable data:

  - mail queue
  - mailman pending confirmations

Agreed. I suspect those queues if they haven't been sent yet are just garbage and should be flushed.

Well, it could be people like us who are migrating mail servers, it's
expected by us to keep trying to deliver those emails...

Probably things like bad email addresses that can't be reached or subscriber spam.

We ALSO have a lot of that, yes, see Making sure you're not a bot!

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

10

On Mon, Sep 15, 2025 at 8:25 PM ‘Sandro Santilli’ via Sac <sac@lists.osgeo.org> wrote:

On Mon, Sep 15, 2025 at 05:10:39PM -0400, Regina Obe wrote:

We are changing the ip to what is currently in the network forward in
osgeo9 but keeping the host name.

Then, when ready, we should re-point all these A records:

tilechache, together with mapserver were on osgeo6, mapserver has been taken care of
https://trac.osgeo.org/osgeo/ticket/3405
tilecache is on osgeo9 osgeo-buster.
What to do with it can be decided later:
https://trac.osgeo.org/osgeo/ticket/3407

That site wasn’t even enabled.
https://gitea.osgeo.org/sac/osgeo9/wiki/mailserver-container#sites-available-cleanup

And update the SPF records to include the new IP (this could be already done now).

I actually wonder what “containerize” really meant

Lxd-migrate is part of lxd source code https://github.com/canonical/lxd/tree/main/lxd-migrate

What I meant was: are we supposed to have an exact copy of all data ?

I think the answer is YES, but from the time lxd-migrate was run the
data on the machines started diverging. How are we going to deal with that ?

I understood there’s an rsync script but that script does NOT copy ALL
the data, just a number of selected subdirectories,

ture

and the script itself
is run from within the new machine, making it impossible to copy (say)
the /etc/cron* directories w/out removing the script.

It’s already been copied, and yes I have modified it,

  • commented out the mailman_stats.sh
  • added the rsync-osgeo6.sh

And when we do the movement, when I remove the rsync-osgeo6.sh I will uncomment the mailman_stats.sh

For other configurations like mailman

perl -pi -e 's/staging\.//' /etc/mailman/mm_cfg.py

Fix lists url if needed: which catch the name on mm_cfg.py
Fix lists permissions if needed. (which have been fixed on osgeo6 and its not needed)

If we cannot account for DNS timing, how do we deal with mail queues ?

We don’t. But as I recall I think mail servers are set to retry for like 72 hours by default if they get a bounce.
So I don’t see this as being an issue except for making sure osgeo6 queue is clear before we switch over.

We’ll need to make sure mail directed at the old IP get a bounce,
to get the retry (hoping the retry will be to the new IP).

As per OUR queues, at the time of writing mail.osgeo.org has 2114 queued messages.
We need to plan how to clear that queue.

Looks like today it has only 219:
root@osgeo6:/home/cvvergara# postqueue -j | wc -l
219

And yes, I dont deny that there will be a disruption on the mail service, a proper announcement is needed.

–strk;

Libre GIS consultant/developer :trumpet:
https://strk.kbt.io/services.html

11

On Tue, Sep 16, 2025 at 01:43:15PM -0600, Vicky Vergara wrote:

On Mon, Sep 15, 2025 at 8:25 PM 'Sandro Santilli' via Sac <
sac@lists.osgeo.org> wrote:

> On Mon, Sep 15, 2025 at 05:10:39PM -0400, Regina Obe wrote:
> >
> > We are changing the ip to what is currently in the network forward in
> > osgeo9 but keeping the host name.
>
> Then, when ready, we should re-point all these A records:
>
> - lists.osgeo.org
> - mail.osgeo.org
> - (*.)tilecache.osgeo.org [ shall we move this ? ]
>

tilechache, together with mapserver were on osgeo6, mapserver has been
taken care of
Making sure you're not a bot!
tilecache is on osgeo9 osgeo-buster.
What to do with it can be decided later:
Making sure you're not a bot!

According to DNS tilecache is expected to be found on whatever host
has IP address 140.211.15.13, which is osgeo9, which proxies ports 80
and 443 to the "nginx" container. In turn, the "nginx" container
proxies requests for "tilecache.org" to the osgeo6-buster container,
so we can say that "tilecache" is on BOTH osgeo9-osgeo6-buster AND
osgeo9-nginx (if you rename the "osgeo6-buster" container you'd also
have to update the nginx proxy.

> - drone.osgeo.org [ shall we drop this ? ]
That site wasn't even enabled.
Making sure you're not a bot!

I've dropped that DNS record already.

> I understood there's an rsync script but that script does NOT copy ALL
> the data, just a number of selected subdirectories,

true

> and the script itself
> is run from within the new machine, making it impossible to copy (say)
> the /etc/cron* directories w/out removing the script.

It's already been copied, and yes I have modified it,

Will it be overridden on next rsync run ?

- commented out the mailman_stats.sh
- added the rsync-osgeo6.sh

-

And when we do the movement, when I remove the rsync-osgeo6.sh I will
uncomment the mailman_stats.sh

For other configurations like mailman

perl -pi -e 's/staging\.//' /etc/mailman/mm_cfg.py

Fix lists url if needed: which catch the name on mm_cfg.py

This part I think we could do with Ansible, did you look at
Making sure you're not a bot! ?
That PR makes it so that the new container is ALSO considered
a production server and thus will use the same default values
for the variables used for the configuration of the mail
and mailman servers. The mm_cfg.py file is deployed from a template
using those variables, see:

  Making sure you're not a bot!

Default variables for the role:

  Making sure you're not a bot!

Fix lists permissions if needed. (which have been fixed on osgeo6 and its not needed)

This would be good to have in ansible too, eventually

> We'll need to make sure mail directed at the old IP get a bounce,
> to get the retry (hoping the retry will be to the new IP).
>
> As per OUR queues, at the time of writing mail.osgeo.org has 2114 queued
> messages.
> We need to plan how to clear that queue.

Looks like today it has only 219:
root@osgeo6:/home/cvvergara# postqueue -j | wc -l
219

That's due to me working on removal of malicious mails,
see Making sure you're not a bot!
Right now there are 198 in the queue, so the fix seemed effective
( we're now refusing mail from <anyone>@service.<anything>.cn )

Among those 198 there are cases of recipient mailbox being full (even
in gmail), unroutable IP addresses or connection refused or hanging
for legit-looking domains like:
  - geospatialvision.com
  - geoaspects.com
  - geostellar.com
  - geologicsystems.com

Those could be real users having temporary difficulties

And yes, I dont deny that there will be a disruption on the mail service,
a proper announcement is needed.

Maybe we can reduce the disruption by moving one service at a time ?
For example, we could start with the Mail Submission Service by
registering a new DNS entry like msa.osgeo.org and ask users of the
service to use that to send mail ?

The mail submission service wiki page is here:
  SAC:Message Submission Agent - OSGeo

Mail submission requires TLS connection thus a valid certificate,
right now we're asking to use "lists.osgeo.org" because at the time
of bringing up the service we didn't know we could have multi-name
certificates, today we can do better (and I've read about your
experiments with adding the "mail.osgeo.org" on that certificate....

Mail submission also requires proper SPF and DKIM records setup,
which we could start working on before moving all the rest
(easily done via Ansible)

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

12

On Wed, Sep 17, 2025 at 4:33 AM Sandro Santilli <strk@kbt.io> wrote:

On Tue, Sep 16, 2025 at 01:43:15PM -0600, Vicky Vergara wrote:

On Mon, Sep 15, 2025 at 8:25 PM ‘Sandro Santilli’ via Sac <
sac@lists.osgeo.org> wrote:

On Mon, Sep 15, 2025 at 05:10:39PM -0400, Regina Obe wrote:

We are changing the ip to what is currently in the network forward in
osgeo9 but keeping the host name.

Then, when ready, we should re-point all these A records:

tilechache, together with mapserver were on osgeo6, mapserver has been
taken care of
https://trac.osgeo.org/osgeo/ticket/3405
tilecache is on osgeo9 osgeo-buster.
What to do with it can be decided later:
https://trac.osgeo.org/osgeo/ticket/3407

According to DNS tilecache is expected to be found on whatever host
has IP address 140.211.15.13, which is osgeo9, which proxies ports 80
and 443 to the “nginx” container. In turn, the “nginx” container
proxies requests for “tilecache.org” to the osgeo6-buster container,
so we can say that “tilecache” is on BOTH osgeo9-osgeo6-buster AND
osgeo9-nginx (if you rename the “osgeo6-buster” container you’d also
have to update the nginx proxy.

osgeo9/osgeo6-buster container is no more, the container name is mail used your suggestion on the chat
osgeo7/osgeo6-buster is still there did not renamed it. And has everything that osgeo6 had before I started looking at the mailserver.
is rsyncing also with osgeo6, therefore has all the mailman data up-to-date

Tilecache is the lease of my concerns:
We have this repository: https://github.com/OSGeo/tilecache Hasn’t been touched in 11 years
Domain is being paid by OSGeo
What to do will depend on the board.
As mentioned in https://trac.osgeo.org/osgeo/ticket/3407

I’ve dropped that DNS record already.

I understood there’s an rsync script but that script does NOT copy ALL
the data, just a number of selected subdirectories,

true

and the script itself
is run from within the new machine, making it impossible to copy (say)
the /etc/cron* directories w/out removing the script.

It’s already been copied, and yes I have modified it,

Will it be overridden on next rsync run ?

The rsync will not override configuration of the cronjob
It does override configuration of the mailboxes
that is why at the moment I am writing this reply I can see Sac mailing list here:
https://lists.staging.osgeo.org/mailman/listinfo
but will not see it after the rsync unless the following command is issued:

/usr/lib/mailman/bin/withlist --lock --run fix_url sac

Because right now mailman configuration is lists.staging
and it’s also not overridden by the rsync

  • commented out the mailman_stats.sh
  • added the rsync-osgeo6.sh

And when we do the movement, when I remove the rsync-osgeo6.sh I will
uncomment the mailman_stats.sh

For other configurations like mailman

perl -pi -e 's/staging\.//' /etc/mailman/mm_cfg.py

That is why the above is needed, to reconfigure mailman to not be lists.staging

Fix lists url if needed: which catch the name on mm_cfg.py

This part I think we could do with Ansible, did you look at
https://gitea.osgeo.org/sac/ansible-deployment/pulls/79 ?
That PR makes it so that the new container is ALSO considered
a production server and thus will use the same default values
for the variables used for the configuration of the mail
and mailman servers. The mm_cfg.py file is deployed from a template
using those variables, see:

https://gitea.osgeo.org/sac/ansible-deployment/src/branch/master/deployment/roles/list-server/templates/etc/mailman/mm_cfg.py.j2#L37

Default variables for the role:

https://gitea.osgeo.org/sac/ansible-deployment/src/branch/master/deployment/roles/list-server/defaults/main/mailman.yml

You can do the ansible commands you have prepared. I do not have problems with that.
But I am not an ansible expert. I do have my petit scripts (some with just echo of commands that I might need to do in the order that have to be done)

Fix lists permissions if needed. (which have been fixed on osgeo6 and its not needed)

This would be good to have in ansible too, eventually

Yeah, not difficult, specially when the list is not created using the list admin page. But not now.

We’ll need to make sure mail directed at the old IP get a bounce,
to get the retry (hoping the retry will be to the new IP).

As per OUR queues, at the time of writing mail.osgeo.org has 2114 queued
messages.
We need to plan how to clear that queue.

Looks like today it has only 219:
root@osgeo6:/home/cvvergara# postqueue -j | wc -l
219

That’s due to me working on removal of malicious mails,
see https://trac.osgeo.org/osgeo/ticket/3437
Right now there are 198 in the queue, so the fix seemed effective
( we’re now refusing mail from @service..cn )

Among those 198 there are cases of recipient mailbox being full (even
in gmail), unroutable IP addresses or connection refused or hanging
for legit-looking domains like:

Those could be real users having temporary difficulties

And yes, I dont deny that there will be a disruption on the mail service,
a proper announcement is needed.

Maybe we can reduce the disruption by moving one service at a time ?
For example, we could start with the Mail Submission Service by
registering a new DNS entry like msa.osgeo.org and ask users of the
service to use that to send mail ?

The mail submission service wiki page is here:
https://wiki.osgeo.org/wiki/SAC:Message_Submission_Agent

Mail submission requires TLS connection thus a valid certificate,
right now we’re asking to use “lists.osgeo.org” because at the time
of bringing up the service we didn’t know we could have multi-name
certificates, today we can do better (and I’ve read about your
experiments with adding the “mail.osgeo.org” on that certificate…

Mail submission also requires proper SPF and DKIM records setup,
which we could start working on before moving all the rest
(easily done via Ansible)

–strk;

Libre GIS consultant/developer :trumpet:
https://strk.kbt.io/services.html

13

On Wed, Sep 17, 2025 at 02:54:21PM -0600, Vicky Vergara wrote:

On Wed, Sep 17, 2025 at 4:33 AM Sandro Santilli <strk@kbt.io> wrote:

> Will it be overridden on next rsync run ?

The rsync will not override configuration of the cronjob
It does override configuration of the mailboxes

Ok that explains why I kept seeing configuration reverted.

Would it be possible for the rsync to ONLY deal with data and
leave configurations untouched ?
I read it should theoretically be enough to copy the data/
archives/ and list/ subdirs (the current rsync is copying all
of /var/lib/mailman instead).

After all we should be almost ready to migrate lists.osgeo.org so
to point to the new container, the only pending things should be:

  - the SPF record

  - ensure the network forward is in place via ansible ?
    [ this can be postponed ]

Is there anything else we need to do before the switch ?

--strk;

  Libre GIS consultant/developer :trumpet:
  strk's services

14

On Thu, Sep 18, 2025 at 10:40 AM Sandro Santilli <strk@kbt.io> wrote:

On Wed, Sep 17, 2025 at 02:54:21PM -0600, Vicky Vergara wrote:

On Wed, Sep 17, 2025 at 4:33 AM Sandro Santilli <strk@kbt.io> wrote:

Will it be overridden on next rsync run ?

The rsync will not override configuration of the cronjob
It does override configuration of the mailboxes

Ok that explains why I kept seeing configuration reverted.

Would it be possible for the rsync to ONLY deal with data and
leave configurations untouched ?

rsync the configuration of the mailboxes is good.
Why would you want me to fix the configuration of mapbender-users and discuss again (remember they were missing the captcha lines)
Why would you want me to process again the password for gdal-dev?
Admins/users change configurations and we need to port those configurations into oosgeo9

For me its ok that the configuration of the mailboxes are the same as on osgeo6

I read it should theoretically be enough to copy the data/
archives/ and list/ subdirs (the current rsync is copying all
of /var/lib/mailman instead).

After all we should be almost ready to migrate lists.osgeo.org so
to point to the new container, the only pending things should be:

  • the SPF record

We are keeping that in mind for the process

  • ensure the network forward is in place via ansible ?
    [ this can be postponed ]

Is there anything else we need to do before the switch ?

–strk;

Libre GIS consultant/developer :trumpet:
https://strk.kbt.io/services.html