[SAC] Mailman migration, backups

Committees SAC mailing list
1

Folks,

I'm making progress on the mailman migration. I have almost
completed a test migration to the VM. It seems rebulding the
HTML archive pages from the master .mbox file is pretty slow!

I'll hopefully announce a downtime period soon and execute the
migration. I will try to minimize the downtime by deferring the
web archive building to a post-processing step. I am tentatively
planning to shutdown the web access (with the intent of preventing
new subscriptions or unsubscriptions) at the start of the window,
migrate things over without html archives, and then repoint DNS to the
new server.

I am *not* currently planning to disable email so there is going to be a
period after snapshotting of the archives and before DNS is pointing to
the new server during which any emails sent to lists do not make it into
real archive. I consider this acceptable lossage, and preferrable to email
bouncing for a few hours. I am willing to consider other opinions on the
matter. Chris' view was different than mine but he was too polite to
push me hard.

Ah yes, back to my main question. As you can imagine I would like to
get backups into place for the new server. I am imagining /etc,
/var/mailman/archives/private/*.mbox and /var/mailman/lists as being
the things to backup. No point in trying to backup all the html files.
I looked at:

  http://wiki.osgeo.org/wiki/SAC:Backups

But I see no information on how to administer Backula on a new VM.
Is there any OSGeo specific information on this available?

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

2

On 10/16/2011 08:29 PM, Frank Warmerdam wrote:

Ah yes, back to my main question. As you can imagine I would like to
get backups into place for the new server. I am imagining /etc,
/var/mailman/archives/private/*.mbox and /var/mailman/lists as being
the things to backup. No point in trying to backup all the html files.
I looked at:

  http://wiki.osgeo.org/wiki/SAC:Backups

But I see no information on how to administer Backula on a new VM.
Is there any OSGeo specific information on this available?

Best regards,

Once the bacula client is installed( I think it already is) and the key
is set so the bacula admin can get in (might also already be done), all
configuration is done on the bacula admin on the Backup server.

I was working on this for the download server, so I'll jot down the
procedure.

Thanks,
Alex

3

On Mon, Oct 17, 2011 at 5:29 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Folks,

I'm making progress on the mailman migration. I have almost
completed a test migration to the VM. It seems rebulding the
HTML archive pages from the master .mbox file is pretty slow!

Frank,

it would be pretty cool to avoid rebulding the HTML archives since
all links to existing emails found in past messages and the various
trac instances etc will become invalid (usually the mailman archive
URLs to messages end up shifted).

Isn't a transfer of the HTML archives with rsync enough?

Markus

4

On 11-10-16 10:16 PM, Markus Neteler wrote:

On Mon, Oct 17, 2011 at 5:29 AM, Frank Warmerdam<warmerdam@pobox.com> wrote:

Folks,

I'm making progress on the mailman migration. I have almost
completed a test migration to the VM. It seems rebulding the
HTML archive pages from the master .mbox file is pretty slow!

Frank,

it would be pretty cool to avoid rebulding the HTML archives since
all links to existing emails found in past messages and the various
trac instances etc will become invalid (usually the mailman archive
URLs to messages end up shifted).

Markus,

It is not immediately clear to me that the archives would be
renumbered when regenerated but it is possible.

Isn't a transfer of the HTML archives with rsync enough?

/me sighs.

I can't even complete a "du" command over the html archives. I am not sure
rsync would ever finish. I suppose I could try it.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/warmerda
and watch the world go round - Rush | Geospatial Software Developer

5

On Sun, Oct 16, 2011 at 10:46:33PM -0700, Frank Warmerdam wrote:

On 11-10-16 10:16 PM, Markus Neteler wrote:
>On Mon, Oct 17, 2011 at 5:29 AM, Frank Warmerdam<warmerdam@pobox.com> wrote:

>>I'm making progress on the mailman migration. I have almost
>>completed a test migration to the VM. It seems rebulding the
>>HTML archive pages from the master .mbox file is pretty slow!

>it would be pretty cool to avoid rebulding the HTML archives since
>all links to existing emails found in past messages and the various
>trac instances etc will become invalid (usually the mailman archive
>URLs to messages end up shifted).

It is not immediately clear to me that the archives would be
renumbered when regenerated but it is possible.

Last time I've migrated a Mailman instance, an old, custom-installed
Mailman on Solaris/Sparc to a more recent one on Debian Linux, just
copying the entire mailman work directory over to the new machine was
the only action required to migrate the data.
If I were to accomplish the migration of the OSGeo Mailman, I wouldn't
expose myself to unneccessary trouble which probably hides behind
rebuilding the archive.

>Isn't a transfer of the HTML archives with rsync enough?

/me sighs.

I can't even complete a "du" command over the html archives. I am not sure
rsync would ever finish. I suppose I could try it.

root@peer1_1:archives # pwd
/var/lib/mailman/archives
root@peer1_1:archives # du -hs *
19G private
20K public

Not a big deal, just requires waiting a few more minutes (we're running
an old-fashioned filesystem on this machine :wink:

You probably want to run 'rsync' at least twice: First to get the big
chunk over and once more to catch those EMails which have arrived in
the meantime. Maybe it's even easier just to build a tarball for the
first chunk. Rsyncing the remaining changes should be a matter of just
a couple of minutes - run a 'find' and 'du' on the directory before to
make sure at least the directory index is in cache. Thus if you make
an announcement, stopping the OSGeo mail service for this period might
be acceptable in order to have everything archived.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------