[SAC] Backups - Especially osgeo1 Mysql for Drupal

Committees SAC mailing list
1

Folks,

I did a bit of a skim of our backups on osgeo1, and I'm not seeing any
current backups being done for the mysql databases. These are most obviously
used for the Drupal instances (osgeo, grass, fdo and mapguide). It seems there
was some sort of zrm (Zmanda) backups being done for mysql at one point, but
it seems it was commented out of /etc/cron.d/backup.cron at some point.

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

   http://wiki.osgeo.org/index.php/SAC:Backups

I also updated it slightly, and referenced it from the SAC Service Status
page.

I have taken the liberty of filing a ticket on the mysql backup issue.
Shawn - perhaps you would be willing to get the zrm stuff going again
and confirm the backups are solid?

In general I think our backup handling is lax/confused/unmonitored and
a review would be good. The wiki is an example of an important service
that is not being backed up as far as I know.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

2

Frank,

Yes. I'll take a look at the mysql backups.

shawn

On 12-Dec-07, at 10:25 AM, Frank Warmerdam wrote:

Folks,

I did a bit of a skim of our backups on osgeo1, and I'm not seeing any
current backups being done for the mysql databases. These are most obviously
used for the Drupal instances (osgeo, grass, fdo and mapguide). It seems there
was some sort of zrm (Zmanda) backups being done for mysql at one point, but
it seems it was commented out of /etc/cron.d/backup.cron at some point.

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

http://wiki.osgeo.org/index.php/SAC:Backups

I also updated it slightly, and referenced it from the SAC Service Status
page.

I have taken the liberty of filing a ticket on the mysql backup issue.
Shawn - perhaps you would be willing to get the zrm stuff going again
and confirm the backups are solid?

In general I think our backup handling is lax/confused/unmonitored and
a review would be good. The wiki is an example of an important service
that is not being backed up as far as I know.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

3

Please note also that we have a few databases to remove from our mysql instance, since things were rearranged during the drupal upgrade. Wolf and I will be working on this - that should help reduce some of the size of the mysql backups too.

Tyler

On 12-Dec-07, at 7:25 AM, Frank Warmerdam wrote:

Folks,

I did a bit of a skim of our backups on osgeo1, and I'm not seeing any
current backups being done for the mysql databases. These are most obviously
used for the Drupal instances (osgeo, grass, fdo and mapguide). It seems there
was some sort of zrm (Zmanda) backups being done for mysql at one point, but
it seems it was commented out of /etc/cron.d/backup.cron at some point.

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

  http://wiki.osgeo.org/index.php/SAC:Backups

I also updated it slightly, and referenced it from the SAC Service Status
page.

I have taken the liberty of filing a ticket on the mysql backup issue.
Shawn - perhaps you would be willing to get the zrm stuff going again
and confirm the backups are solid?

In general I think our backup handling is lax/confused/unmonitored and
a review would be good. The wiki is an example of an important service
that is not being backed up as far as I know.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Tyler Mitchell
Executive Director
Open Source Geospatial Foundation
tmitchell@osgeo.org
P: +1-250-277-1621
M: +1-250-303-1831

4

On 12-Dec-07, at 7:25 AM, Frank Warmerdam wrote:

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

  http://wiki.osgeo.org/index.php/SAC:Backups

I added a section on that page for listing the folders to be backed up. Also I opened a new ticket to trac our discussion about defining the backup locations/folders for PEER1.

http://trac.osgeo.org/osgeo/ticket/179

5

Tyler Mitchell (OSGeo) wrote:

On 12-Dec-07, at 7:25 AM, Frank Warmerdam wrote:

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

  http://wiki.osgeo.org/index.php/SAC:Backups

I added a section on that page for listing the folders to be backed up. Also I opened a new ticket to trac our discussion about defining the backup locations/folders for PEER1.

http://trac.osgeo.org/osgeo/ticket/179

Tyler,

I've noted that on osgeo1 we should at least backup:

# /etc - 10MB
# /var/www/trac - 550MB
# /root - 3MB
# /var/lib/mailman/lists - 36MB

We really do want to backup /var/lib/mailman/archives/private/*.mbox but
I'm not sure how easy that is to specify. We don't really want all the
htmlified archives also under /var/lib/mailman/archives as this is a great
deal of additional bulk, and scanning all these tiny files is likely to be
painful from a system load point of view.

Actually, it looks like /home/back/mailman/archives is what we want for
mailman configuration, but it does not appear to be getting updated. All
the mbox files date from early 2007 or 2006. I'll look into this a bit.

Howard has indicated that the svn mirroring should be sufficient for
those purposes.

I looked in /var/www/html and there is a whack of stuff I don't think we
need to backup (ie. all the fdo and mapguide docs). On the other hand
there is likely stuff there that should be backed up. I'm not sure how
we want to handle this.

Boy, /home/back is a mess. I'll dig some more.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

6

Thanks a lot for digging into it Frank!

I'll be back in the office chair on the 3rd.

Tyler

On 29-Dec-07, at 4:44 PM, Frank Warmerdam wrote:

Tyler Mitchell (OSGeo) wrote:

On 12-Dec-07, at 7:25 AM, Frank Warmerdam wrote:

It is a bit hard to figure out what is what with regard to backups, so I have
taken the liberty of extracting the backup portion out of the Migration
Documentation wiki page Shawn wrote, and setting it up as SAC:Backups:

  http://wiki.osgeo.org/index.php/SAC:Backups

I added a section on that page for listing the folders to be backed up. Also I opened a new ticket to trac our discussion about defining the backup locations/folders for PEER1.
http://trac.osgeo.org/osgeo/ticket/179

Tyler,

I've noted that on osgeo1 we should at least backup:

# /etc - 10MB
# /var/www/trac - 550MB
# /root - 3MB
# /var/lib/mailman/lists - 36MB

We really do want to backup /var/lib/mailman/archives/private/*.mbox but
I'm not sure how easy that is to specify. We don't really want all the
htmlified archives also under /var/lib/mailman/archives as this is a great
deal of additional bulk, and scanning all these tiny files is likely to be
painful from a system load point of view.

Actually, it looks like /home/back/mailman/archives is what we want for
mailman configuration, but it does not appear to be getting updated. All
the mbox files date from early 2007 or 2006. I'll look into this a bit.

Howard has indicated that the svn mirroring should be sufficient for
those purposes.

I looked in /var/www/html and there is a whack of stuff I don't think we
need to backup (ie. all the fdo and mapguide docs). On the other hand
there is likely stuff there that should be backed up. I'm not sure how
we want to handle this.

Boy, /home/back is a mess. I'll dig some more.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | President OSGeo, http://osgeo.org

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Tyler Mitchell
Executive Director
Open Source Geospatial Foundation
tmitchell@osgeo.org
P: +1-250-277-1621
M: +1-250-303-1831

7

A few important questions that I could use an opinion on sooner than later...

I gave our list of folders to include to PEER1 but they came back asking for which folders to _exclude_ not include. I wish we could just say ignore everything except /home/back. This is somewhat how Shawn set it up initially anyway. Can we make this possible by symlinking in all required folders into there?

The other point with their new system is that we have a 50GB limit, but that includes daily incrementals with storing of *6* past revisions of each file. The trouble is we can't change this option, which I expressed was a very poor value for money for us because it ends up meaning we have way less than 50GB of real/current files able to be backed up. The other option is to switch to WEEKLY backups, where they will maintain only the latest version of each file. This will make it much easier for us to manage and estimate our backup usage.

Should we keep daily or go for weekly?

Perhaps important daily/incrementals can be rsync'd to osgeo2 or telascience and vice versa instead of using their service. I'm fairly disillusioned about the service and if we had an alternative, I'd suggest dropping the service.

I'd love to hear your thoughts, I'm trying to iron out our backup service with them this week.

Tyler

8

Folks,

I propose that we convene an IRC meeting Tuesday January 8th at 10 AM Central US (1600 GMT) on #telascience to discuss and decide our backup strategies for osgeo1 and osgeo2. I hope that all primary administrators and other interested parties can attend.

Howard

9

On Thu, Jan 03, 2008 at 03:49:07PM -0600, Howard Butler wrote:

I propose that we convene an IRC meeting Tuesday January 8th at 10 AM
Central US (1600 GMT) on #telascience to discuss and decide our backup
strategies for osgeo1 and osgeo2. I hope that all primary
administrators and other interested parties can attend.

Yup, 1600 UTC should be fine,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

10

I can make it too.

On 3-Jan-08, at 1:49 PM, Howard Butler wrote:

Folks,

I propose that we convene an IRC meeting Tuesday January 8th at 10 AM Central US (1600 GMT) on #telascience to discuss and decide our backup strategies for osgeo1 and osgeo2. I hope that all primary administrators and other interested parties can attend.

Howard

11

It appears that peer1 rebooted us about four hours ago. Were we ever given any notification that this would happen? Tyler? The LDAP had crashed tables and was manually recovered. Please note any other services that have not come back as expected.

/me grumbles at our hosting provider. Every one of them sucks, just in different ways, right?

Howard

12

i can make the meeting

shawn

On 3-Jan-08, at 4:49 PM, Howard Butler wrote:

Folks,

I propose that we convene an IRC meeting Tuesday January 8th at 10 AM Central US (1600 GMT) on #telascience to discuss and decide our backup strategies for osgeo1 and osgeo2. I hope that all primary administrators and other interested parties can attend.

Howard
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

13

I have no idea what happened. It sounds like it happened a couple hours after I (re)submitted my backup changes to tech support, but otherwise I can't think of any reasons. Where is the log trail? We could submit report about the incident.

Tyler

On 3-Jan-08, at 8:49 PM, Howard Butler wrote:

It appears that peer1 rebooted us about four hours ago. Were we ever given any notification that this would happen? Tyler? The LDAP had crashed tables and was manually recovered. Please note any other services that have not come back as expected.

/me grumbles at our hosting provider. Every one of them sucks, just in different ways, right?

Howard

14

Hi guys,

On Thu, Jan 03, 2008 at 03:49:07PM -0600, Howard Butler wrote:

I propose that we convene an IRC meeting Tuesday January 8th at 10 AM
Central US (1600 GMT) on #telascience to discuss and decide our backup
strategies for osgeo1 and osgeo2. I hope that all primary
administrators and other interested parties can attend.

I'm very sorry that I miserably failed to attend the meeting. I simply
was so much involved in what I was doing on tuesday afternoon that I
completely forgot about our 'date'.
Would anyone please be so kind and put into one or two phrases what
you've actually been deciding on tuesday ? Any 'urgent' jobs to do ?

Again, sorry,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------