I would like to start a thread where folks who are primarily responsible for some piece of our infrastructure describe the current state of the backup situation of that component. It would be instructive to find out where we are with things, and it would help illuminate work items that we need to complete. I think the following components need to be described, and I'll start with subversion:
- subversion
- Trac
- wiki
- drupal
- buildbot
- download.osgeo.org
- upload.osgeo.org
- apache configuration
- LDAP
- mailing lists
...
Subversion
---------------------
Subversion's primary backup is an offsite, svnsync server at http://svnmirror.osgeo.org. As new repositories are added to osgeo1, new mirrors are also manually created by Howard at svnmirror.osgeo.org. svnmirror is housed on a server at TelaScience. Where possible, the GUID of the repositories that we manage on osgeo1 are the same on the mirror. Some repositories that are not housed at osgeo1 are also in the mirror, such as TileCache, FeatureServer, and OpenLayers, as a courtesy to these projects.
An incremental dump of subversion is also available for some repositories at http://svn.osgeo.org/dumps, and this directory is wget --mirror'd to a machine at TelaScience. This approach was done before the svnsync stuff was in wide use. It is not known if recovery from these incremental dumps is practically possible, but it is theoretically feasible. The standard svn mirror makes the issue moot, however.
If a catastrophic failure were to happen today, how screwed would we be?
------------------------------------------------------------------------------------------------------------
The mirror does a sync at 19 minutes past the hour, every hour. Worst case, our data would be one hour old, and individuals would also have their local copies of various trees that they could use to commit missing revisions that slipped away in that hour.
A major component of subversion is also the Apache configuration and the hook scripts. At this time, it is not known if/how the hook scripts are backed up. The Apache configuration is backed up into a subversion repository that is locally hosted on osgeo1, but that repo is not mirrored onto the svnmirror.osgeo.org mirror because of its sensitive nature.
Action Items
............................
- Determine how the hook scripts are backed up and ensure that they and their configuration for each repository is preserved.
- Determine how to better offsite the Apache configuration
Recovery procedure from a catastrophic event
-------------------------------------------------------------
- Notify Howard to log into svnmirror.osgeo.org and create dumps of all repositories
- Import those dumps into the new home
- Re-enable Apache configuration from backup
- Re-enable hook scripts from backup