[SAC] Backup and migration ideas

Committees SAC mailing list
1

Hi all,
There are a couple systems related ideas on my mind for a while and I wanted to put them past you for your thoughts, ideas, critique. In general, I've been digging into our options for hosting at http://www.osuosl.org and a few ways we could capitalise on the opportunity without making too much work in the short term.

1 - Migrate services from osgeo2 to OSL.
This is likely the simplest server to move over and has the added benefit of reducing our costs by around 50%. In some ways we've maxed out osgeo2 and upgrading is not cheap. I suggest we focus on building on a new secondary server
* Systems affected: wiki, planet, rsync backups from other servers, development Drupal sites, others?
* Challenges: LDAP auth back to osgeo1 needed?, others?

2 - Mirror project websites onto OSL.
If successful, then potentially switch over to the new server where appropriate. My particular interest here is to have all main OSGeo project websites served in a managed server environment. Hosting these onto osgeo1 typically opens up security questions. We may be able to get VMs for each project or use LDAP shell authentication (like on some xblades) to handle the various projects in the same VM. Could also open this up to include projects not hosted on OSGeo infrastructure, even just static mirrors.
* Sites affected: gdal, grass, qgis, geotools, mapserver (?), geonetwork, mapbender, ?
* Challenges: several different platforms, shell authentication, ?

3- Build upgraded osgeo.org site on OSL machine.
Wolf and I have discussed a Drupal upgrade over the next few months that should fix a few things. Building it on a new system has some advantages when upgrading all our modules, etc. I already have an account and Drupal set up on an OSL machine for testing.
* Sites affected: www.osgeo.org, mapguide, fdo, mapbender
* Challenges: tbd

This is probably enough for now. Here's my summary:
#1 is less critical but can save some money right away. It's moderately difficult to complete but fortunately is a small set of services.
  #2 is the most critical in my mind. It may also be the hardest to implement, needing a pretty clearly laid out set of dependencies and some get sysadmin assistance. Depending on how it's realised, some project leads may be able to handle their own needs if we get the relationship with OSL set up well enough (and have LDAP auth stuff working from there)
#3 is already pretty much set up and will be on Wolf and I's plate to handle anyway.

Anyone interested in helping or pursuing or debating these? I'm mainly just brainstorming, but trying to envision some compartmentalized ideas out for discussion. Obviously we'd need owners for each part. The value I see in doing some of the switches is worth it for me to help, but obviously I cannot maintain it over the long term. Perhaps others might like to join SAC to help see some of these through if there isn't enough volunteer power??

Looking forward to your thoughts,
Tyler

2

On Thu, Aug 20, 2009 at 22:15, Tyler Mitchell
(OSGeo)<tmitchell@osgeo.org> wrote:

1 - Migrate services from osgeo2 to OSL.
2 - Mirror project websites onto OSL.
3- Build upgraded osgeo.org site on OSL machine.

I agree with Tyler. #2 is the most imporant at this stage. #1 Should,
I think, give us access to more modern software. The old servers are
running quite old versions of all programs and no-one has really a
clue on how to upgrade.

#3 is perhaps the most difficult because it has the oddest, quirkiest
apache setup I have ever seen. Probably use to the * domain name
handling. Perhaps we could get rid of this at the same time, and clean
up the apache configs so that new websites can be more easily managed.
As it is now no-one seems to feel comfortable fiddling with them.
Drupal6 will indeed bring in many improvements.

Anyone interested in helping or pursuing or debating these? I'm mainly just
brainstorming, but trying to envision some compartmentalized ideas out for
discussion. Obviously we'd need owners for each part. The value I see in
doing some of the switches is worth it for me to help, but obviously I
cannot maintain it over the long term. Perhaps others might like to join
SAC to help see some of these through if there isn't enough volunteer
power??

Needless to say I will gladly help with #3, but I'd also like to help
with #2, at least for the GRASS GIS part. Perhaps we could find one
volunteer from all projects affected to help us out with "their" site?

--Wolf

--
<8 )---- Wolf Bergenheim ----( 3>

3

On Thu, Aug 20, 2009 at 9:27 PM, Wolf
Bergenheim<wolf+grass@bergenheim.net> wrote:

On Thu, Aug 20, 2009 at 22:15, Tyler Mitchell
(OSGeo)<tmitchell@osgeo.org> wrote:

1 - Migrate services from osgeo2 to OSL.
2 - Mirror project websites onto OSL.
3- Build upgraded osgeo.org site on OSL machine.

...

Needless to say I will gladly help with #3, but I'd also like to help
with #2, at least for the GRASS GIS part. Perhaps we could find one
volunteer from all projects affected to help us out with "their" site?

Count me in.

Markus

4

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a thorn in our side since we went off on our own -- a holdover from collabnet. Anything that's important has its own DNS entry now, right?

5

On Aug 20, 2009, at 2:15 PM, Tyler Mitchell (OSGeo) wrote:

Hi all,
There are a couple systems related ideas on my mind for a while and I wanted to put them past you for your thoughts, ideas, critique. In general, I've been digging into our options for hosting at http://www.osuosl.org and a few ways we could capitalise on the opportunity without making too much work in the short term.

1 - Migrate services from osgeo2 to OSL.
This is likely the simplest server to move over and has the added benefit of reducing our costs by around 50%. In some ways we've maxed out osgeo2 and upgrading is not cheap. I suggest we focus on building on a new secondary server
* Systems affected: wiki, planet, rsync backups from other servers, development Drupal sites, others?
* Challenges: LDAP auth back to osgeo1 needed?, others?

I fully support this as a way to dip our toes into OSL. I'm in no position to lead such an effort, but I'm willing to help bikeshed. Having each service virtualized with whomever does the work's choice of virtualization environment would be fantastic. osgeo2 and osgeo1 don't get upgraded at all because the rigor of interdependencies makes the prospect quite scary. If we can have more independent chunks, we can break smaller things when we do stuff.

2 - Mirror project websites onto OSL.
If successful, then potentially switch over to the new server where appropriate. My particular interest here is to have all main OSGeo project websites served in a managed server environment. Hosting these onto osgeo1 typically opens up security questions. We may be able to get VMs for each project or use LDAP shell authentication (like on some xblades) to handle the various projects in the same VM. Could also open this up to include projects not hosted on OSGeo infrastructure, even just static mirrors.
* Sites affected: gdal, grass, qgis, geotools, mapserver (?), geonetwork, mapbender, ?
* Challenges: several different platforms, shell authentication, ?

Project sites are on the blades because this is where we feel most comfortable giving them the keys and letting them drive off their own cliffs. A virtualization effort, similar to above (on the same box even?) that allows each project to have their own keys would be enticing enough for people to go through the pain of a server move. Some of projects, like MapServer, GDAL, and GRASS were able to mirror off to other places in the time it took for DNS to switch over during the past outage. Other projects weren't quite as ready as that, but that's always been the deal with running on the blades.

3- Build upgraded osgeo.org site on OSL machine.
Wolf and I have discussed a Drupal upgrade over the next few months that should fix a few things. Building it on a new system has some advantages when upgrading all our modules, etc. I already have an account and Drupal set up on an OSL machine for testing.
* Sites affected: www.osgeo.org, mapguide, fdo, mapbender

... subversion, trac, postgresql for trac, mysql, ldap. There's a ton of critical stuff on osgeo1 right now. svn/trac + its db should probably be separated onto its own instance somewhere (it also probably best-positioned and most self-contained for a move off of osgeo1 as well).

6

On Fri, Aug 21, 2009 at 06:00, Howard Butler<hobu.inc@gmail.com> wrote:

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a thorn in
our side since we went off on our own -- a holdover from collabnet.
Anything that's important has its own DNS entry now, right?

I agree :). And we should be able to see from the logs which ones are used.

--Wolf

--
<8 )---- Wolf Bergenheim ----( 3>

7

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a
thorn in our side since we went off on our own -- a holdover from
collabnet. Anything that's important has its own DNS entry now, right?

Sounds good to me too - perhaps a few of us could take on shifts to
monitor its continued usage, if any.

8

Thanks for the feedback guys, I'm glad it's not totally off the rocker.
One more important point to add...

I've been pinging OSL on and off on similar ideas the past few weeks.
Using the existing VM they gave us would be simple for the Drupal upgrade,
but beyond that they JUST let me know they are hurting for hardware space
at the moment. So, I'm not sure what they were thinking when discussing
hosting ideas with us at OSCON, but either way...

I will therefore propose that we get some nice new hardware and host it
there. Anyone on this list especially good at spec'ing out machines? I
assume we can get some good hardware for the equivalent cost of a couple
months (~$3k) of our current hosting and shouldn't make a major impact on
our overall finances, especially if we can end up retiring osgeo2.

I can see that using the blades has already helped us get into the mindset
of using VMs for compartmentalising our sets of services into groups. Now
this is where I'm beyond my knowledge to recommend anything, but setting
up virtual server instances is something that OSL staff seem to do quite
well. I'll start up some docs to help brainstorm this general migration
discussion and then we'll have something to take up with them when we are
ready to dig deeper.

Thanks again for the comments, but especially for your continued help with
OSGeo systems. It's nice to see that outages are so rare that we really
notice them :wink:

Tyler

On Aug 20, 2009, at 2:15 PM, Tyler Mitchell (OSGeo) wrote:

Hi all,
There are a couple systems related ideas on my mind for a while and
I wanted to put them past you for your thoughts, ideas, critique.
In general, I've been digging into our options for hosting at
http://www.osuosl.org
and a few ways we could capitalise on the opportunity without
making too much work in the short term.

1 - Migrate services from osgeo2 to OSL.
This is likely the simplest server to move over and has the added
benefit of reducing our costs by around 50%. In some ways we've
maxed out osgeo2 and upgrading is not cheap. I suggest we focus on
building on a new secondary server
* Systems affected: wiki, planet, rsync backups from other servers,
development Drupal sites, others?
* Challenges: LDAP auth back to osgeo1 needed?, others?

I fully support this as a way to dip our toes into OSL. I'm in no
position to lead such an effort, but I'm willing to help bikeshed.
Having each service virtualized with whomever does the work's choice
of virtualization environment would be fantastic. osgeo2 and osgeo1
don't get upgraded at all because the rigor of interdependencies makes
the prospect quite scary. If we can have more independent chunks, we
can break smaller things when we do stuff.

2 - Mirror project websites onto OSL.
If successful, then potentially switch over to the new server where
appropriate. My particular interest here is to have all main OSGeo
project websites served in a managed server environment. Hosting
these onto osgeo1 typically opens up security questions. We may be
able to get VMs for each project or use LDAP shell authentication
(like on some xblades) to handle the various projects in the same
VM. Could also open this up to include projects not hosted on OSGeo
infrastructure, even just static mirrors.
* Sites affected: gdal, grass, qgis, geotools, mapserver (?),
geonetwork, mapbender, ?
* Challenges: several different platforms, shell authentication, ?

Project sites are on the blades because this is where we feel most
comfortable giving them the keys and letting them drive off their own
cliffs. A virtualization effort, similar to above (on the same box
even?) that allows each project to have their own keys would be
enticing enough for people to go through the pain of a server move.
Some of projects, like MapServer, GDAL, and GRASS were able to mirror
off to other places in the time it took for DNS to switch over during
the past outage. Other projects weren't quite as ready as that, but
that's always been the deal with running on the blades.

3- Build upgraded osgeo.org site on OSL machine.
Wolf and I have discussed a Drupal upgrade over the next few months
that should fix a few things. Building it on a new system has some
advantages when upgrading all our modules, etc. I already have an
account and Drupal set up on an OSL machine for testing.
* Sites affected: www.osgeo.org, mapguide, fdo, mapbender

... subversion, trac, postgresql for trac, mysql, ldap. There's a ton
of critical stuff on osgeo1 right now. svn/trac + its db should
probably be separated onto its own instance somewhere (it also
probably best-positioned and most self-contained for a move off of
osgeo1 as well).

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

9

Howard Butler wrote:

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a thorn in our side since we went off on our own -- a holdover from collabnet. Anything that's important has its own DNS entry now, right?

Folks,

I agree with turning it off and monitoring for problems.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

10

On Aug 21, 2009, at 8:28 AM, Frank Warmerdam wrote:

Howard Butler wrote:

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a thorn in our side since we went off on our own -- a holdover from collabnet. Anything that's important has its own DNS entry now, right?

Folks,

I agree with turning it off and monitoring for problems.

ok, I have removed it. it was pointed at .242 in case we need to bring it back :wink:

Howard

11

Howard Butler wrote:

On Aug 21, 2009, at 8:28 AM, Frank Warmerdam wrote:

Howard Butler wrote:

On Aug 20, 2009, at 2:27 PM, Wolf Bergenheim wrote:

Probably use to the * domain name handling.

Let's turn the wildcard off now and see what breaks. It has been a thorn in our side since we went off on our own -- a holdover from collabnet. Anything that's important has its own DNS entry now, right?

Folks,

I agree with turning it off and monitoring for problems.

ok, I have removed it. it was pointed at .242 in case we need to bring it back :wink:

Howard,

I discovered a bunch of domain forwardings were failing - ones in the
/etc/httpd/conf.d/rewrite.conf file. I have added explicit DNS A records
for them all.

Stuff like http://osgeo4w.osgeo.org, and http://fr.osgeo.org/

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent