RE: [SAC] MapGuide SVN Down Again

Did MapBender get migrated to the alternate data store? Did it work OK?

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Robert Bray
Sent: Wednesday, February 14, 2007 15:57
To: System Administration Committee Discussion/OSGeo
Subject: [SAC] MapGuide SVN Down Again
Importance: High

2nd time today...help!

Thanks,
Bob
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Fixed.

After some investigation, it appears that the backup scripts are causing the issue. We had backup scripts being run out of /home/sbarnes/scripts that were hung, then root came along and ran its own (and hung). It appears interaction of these scripts is locking the repository and causing our issues.

Shawn,

Can you investigate why crons might still be running out of your home directory? Here's what the process table looked like when I checked earlier tonight:

[hobu@osgeo1 repos]$ ps aux|grep svn
root 8553 0.0 0.0 2280 880 ? Ss 15:45 0:00 /bin/bash /root/scripts/backup_trac_svn.sh
root 8776 0.0 0.0 3084 868 ? S 16:01 0:00 sh -c svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide > "/home/back/svn_backup/2007-02-14_15.45.02"/mapguide
root 8777 0.0 0.1 8628 2688 ? S 16:01 0:00 svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
root 11503 0.0 0.0 2792 868 ? Ss 18:45 0:00 /bin/bash /root/scripts/backup_trac_svn.sh
root 11504 0.0 0.1 4460 2312 ? S 18:45 0:00 perl /home/sbarnes/scripts/svn_repos_backup.pl
root 11907 0.0 0.0 2756 868 ? S 19:01 0:00 sh -c svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide > "/home/back/svn_backup/2007-02-14_18.45.01"/mapguide
root 11908 0.0 0.1 9416 2792 ? S 19:01 0:00 svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
hobu 13533 0.0 0.0 5616 652 pts/0 S+ 20:58 0:00 grep svn

I think some of our repositories are quite large to be doing full dumps as frequently as we are (~8 times per day). Gzipped, the mapguide repository is ~710mb, and doing a full delta dump of probably takes quite a while and is stepping on incoming commits.

I propose change things to do incrementals throughout the day at the same frequency we are currently doing them and a single full backup outside of peak development hours. Someone please propose a good UTC time to do a full dump so as not to step on folks. We are worldly enough to make this somewhat hard :wink:

Howard

On Feb 14, 2007, at 6:09 PM, Jason Birch wrote:

Did MapBender get migrated to the alternate data store? Did it work OK?

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Robert Bray
Sent: Wednesday, February 14, 2007 15:57
To: System Administration Committee Discussion/OSGeo
Subject: [SAC] MapGuide SVN Down Again
Importance: High

2nd time today...help!

Thanks,
Bob
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
Sac Info Page
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
Sac Info Page

On 14-Feb-07, at 5:19 PM, Howard Butler wrote:

Fixed.

After some investigation, it appears that the backup scripts are causing the issue. We had backup scripts being run out of /home/sbarnes/scripts that were hung, then root came along and ran its own (and hung). It appears interaction of these scripts is locking the repository and causing our issues.

Good sleuthing!

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Have you looked at using svnadmin hotcopy? Per the manual:
"You can run this command at any time and make a safe copy of the repository, regardless of whether other processes are using the repository."

- -gary

On Feb 14, 2007, at 4:19 PM, Howard Butler wrote:

Fixed.

After some investigation, it appears that the backup scripts are causing the issue. We had backup scripts being run out of /home/sbarnes/scripts that were hung, then root came along and ran its own (and hung). It appears interaction of these scripts is locking the repository and causing our issues.

Shawn,

Can you investigate why crons might still be running out of your home directory? Here's what the process table looked like when I checked earlier tonight:

[hobu@osgeo1 repos]$ ps aux|grep svn
root 8553 0.0 0.0 2280 880 ? Ss 15:45 0:00 /bin/bash /root/scripts/backup_trac_svn.sh
root 8776 0.0 0.0 3084 868 ? S 16:01 0:00 sh -c svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide > "/home/back/svn_backup/2007-02-14_15.45.02"/mapguide
root 8777 0.0 0.1 8628 2688 ? S 16:01 0:00 svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
root 11503 0.0 0.0 2792 868 ? Ss 18:45 0:00 /bin/bash /root/scripts/backup_trac_svn.sh
root 11504 0.0 0.1 4460 2312 ? S 18:45 0:00 perl /home/sbarnes/scripts/svn_repos_backup.pl
root 11907 0.0 0.0 2756 868 ? S 19:01 0:00 sh -c svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide > "/home/back/svn_backup/2007-02-14_18.45.01"/mapguide
root 11908 0.0 0.1 9416 2792 ? S 19:01 0:00 svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
hobu 13533 0.0 0.0 5616 652 pts/0 S+ 20:58 0:00 grep svn

I think some of our repositories are quite large to be doing full dumps as frequently as we are (~8 times per day). Gzipped, the mapguide repository is ~710mb, and doing a full delta dump of probably takes quite a while and is stepping on incoming commits.

I propose change things to do incrementals throughout the day at the same frequency we are currently doing them and a single full backup outside of peak development hours. Someone please propose a good UTC time to do a full dump so as not to step on folks. We are worldly enough to make this somewhat hard :wink:

Howard

On Feb 14, 2007, at 6:09 PM, Jason Birch wrote:

Did MapBender get migrated to the alternate data store? Did it work OK?

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Robert Bray
Sent: Wednesday, February 14, 2007 15:57
To: System Administration Committee Discussion/OSGeo
Subject: [SAC] MapGuide SVN Down Again
Importance: High

2nd time today...help!

Thanks,
Bob
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

- -_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
Gary Sherman
Micro Resources: http://mrcc.com
   *Geospatial Hosting
   *Web Site Hosting
"We work virtually everywhere"
- -_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFF08Ak1zKuzV6goTgRAq6CAJ0VcYMAWviPquGH+u7O2BvHrLQrRgCfbFWC
+gfj4/NrzuUEoxlBXvW6R4g=
=8y5r
-----END PGP SIGNATURE-----

Yes, i'll look into this right now.

Howard Butler wrote:

Fixed.

After some investigation, it appears that the backup scripts are causing
the issue. We had backup scripts being run out of /home/sbarnes/scripts
that were hung, then root came along and ran its own (and hung). It
appears interaction of these scripts is locking the repository and
causing our issues.

Shawn,

Can you investigate why crons might still be running out of your home
directory? Here's what the process table looked like when I checked
earlier tonight:

[hobu@osgeo1 repos]$ ps aux|grep svn
root 8553 0.0 0.0 2280 880 ? Ss 15:45 0:00
/bin/bash /root/scripts/backup_trac_svn.sh
root 8776 0.0 0.0 3084 868 ? S 16:01 0:00 sh -c
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide >
"/home/back/svn_backup/2007-02-14_15.45.02"/mapguide
root 8777 0.0 0.1 8628 2688 ? S 16:01 0:00
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
root 11503 0.0 0.0 2792 868 ? Ss 18:45 0:00
/bin/bash /root/scripts/backup_trac_svn.sh
root 11504 0.0 0.1 4460 2312 ? S 18:45 0:00 perl
/home/sbarnes/scripts/svn_repos_backup.pl
root 11907 0.0 0.0 2756 868 ? S 19:01 0:00 sh -c
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide >
"/home/back/svn_backup/2007-02-14_18.45.01"/mapguide
root 11908 0.0 0.1 9416 2792 ? S 19:01 0:00
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
hobu 13533 0.0 0.0 5616 652 pts/0 S+ 20:58 0:00 grep svn

I think some of our repositories are quite large to be doing full dumps
as frequently as we are (~8 times per day). Gzipped, the mapguide
repository is ~710mb, and doing a full delta dump of probably takes
quite a while and is stepping on incoming commits.

I propose change things to do incrementals throughout the day at the
same frequency we are currently doing them and a single full backup
outside of peak development hours. Someone please propose a good UTC
time to do a full dump so as not to step on folks. We are worldly
enough to make this somewhat hard :wink:

Howard

On Feb 14, 2007, at 6:09 PM, Jason Birch wrote:

Did MapBender get migrated to the alternate data store? Did it work OK?

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Robert Bray
Sent: Wednesday, February 14, 2007 15:57
To: System Administration Committee Discussion/OSGeo
Subject: [SAC] MapGuide SVN Down Again
Importance: High

2nd time today...help!

Thanks,
Bob
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

I've fixed the script to not use the backup scripts in ~/sbarnes

I've ran tests with the same setup on a test box i have using the same
scripts for backup and the bdb doesn't cause a lock that needs a recover
or hang. But, yes the dumps do take quite a bit of time to complete.

Definitely an incremental backup is the better way to backup the
repositories.

shawn

shawn barnes wrote:

Yes, i'll look into this right now.

Howard Butler wrote:

Fixed.

After some investigation, it appears that the backup scripts are causing
the issue. We had backup scripts being run out of /home/sbarnes/scripts
that were hung, then root came along and ran its own (and hung). It
appears interaction of these scripts is locking the repository and
causing our issues.

Shawn,

Can you investigate why crons might still be running out of your home
directory? Here's what the process table looked like when I checked
earlier tonight:

[hobu@osgeo1 repos]$ ps aux|grep svn
root 8553 0.0 0.0 2280 880 ? Ss 15:45 0:00
/bin/bash /root/scripts/backup_trac_svn.sh
root 8776 0.0 0.0 3084 868 ? S 16:01 0:00 sh -c
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide >
"/home/back/svn_backup/2007-02-14_15.45.02"/mapguide
root 8777 0.0 0.1 8628 2688 ? S 16:01 0:00
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
root 11503 0.0 0.0 2792 868 ? Ss 18:45 0:00
/bin/bash /root/scripts/backup_trac_svn.sh
root 11504 0.0 0.1 4460 2312 ? S 18:45 0:00 perl
/home/sbarnes/scripts/svn_repos_backup.pl
root 11907 0.0 0.0 2756 868 ? S 19:01 0:00 sh -c
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide >
"/home/back/svn_backup/2007-02-14_18.45.01"/mapguide
root 11908 0.0 0.1 9416 2792 ? S 19:01 0:00
svnadmin dump --quiet --deltas /var/www/svn/repos/mapguide
hobu 13533 0.0 0.0 5616 652 pts/0 S+ 20:58 0:00 grep svn

I think some of our repositories are quite large to be doing full dumps
as frequently as we are (~8 times per day). Gzipped, the mapguide
repository is ~710mb, and doing a full delta dump of probably takes
quite a while and is stepping on incoming commits.

I propose change things to do incrementals throughout the day at the
same frequency we are currently doing them and a single full backup
outside of peak development hours. Someone please propose a good UTC
time to do a full dump so as not to step on folks. We are worldly
enough to make this somewhat hard :wink:

Howard

On Feb 14, 2007, at 6:09 PM, Jason Birch wrote:

Did MapBender get migrated to the alternate data store? Did it work OK?

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org]
On Behalf Of Robert Bray
Sent: Wednesday, February 14, 2007 15:57
To: System Administration Committee Discussion/OSGeo
Subject: [SAC] MapGuide SVN Down Again
Importance: High

2nd time today...help!

Thanks,
Bob
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac