We're currently stuck in a weird spot with trying to upgrade to Debian
7, a relic of the original VM disk layout that doesn't get along with GRUB2.
The fastest way to get there seems to be to do a new Base VM based on
debian 7 and clone it. This would be a good idea if we want to start
making smaller VMs to split apart some services into more isolation.
On the one hand its more VMs or containers to deal with, on the other it
might bring happy faces and stability to more things.
Our other option is some hackery to try and make current Vms upgradeable.
Currently the only requests I've heard for a new VM is from QGIS who
wants to remove their cruft from the existing VM by starting fresh
(since they moved all but 2 sites to another server elsewhere anyways).
I wouldn't be opposed to some others moving off of Projects in a similar
manner. We can worry about the main osgeo services separately.
This of course leaves some questions:
Who can help setup the new base vm?
Should we implement puppet or chef, etc to make managing more servers
easier?
Should we consider using LXC containers to isolate projects from each
other on a common kernel?
Is this a good time to remind everyone of the
heavily-used-yet-not-supported AdhocVM? http://wiki.osgeo.org/wiki/AdhocVM Several projects rely on this VM
heavily, yet there are no backups etc. This has been reminded to me
recently by another community member. So, I'm nudging
-jeff
On 2014-05-09, 4:20 PM, Alex Mandel wrote:
We're currently stuck in a weird spot with trying to upgrade to Debian
7, a relic of the original VM disk layout that doesn't get along with GRUB2.
The fastest way to get there seems to be to do a new Base VM based on
debian 7 and clone it. This would be a good idea if we want to start
making smaller VMs to split apart some services into more isolation.
On the one hand its more VMs or containers to deal with, on the other it
might bring happy faces and stability to more things.
Our other option is some hackery to try and make current Vms upgradeable.
Currently the only requests I've heard for a new VM is from QGIS who
wants to remove their cruft from the existing VM by starting fresh
(since they moved all but 2 sites to another server elsewhere anyways).
I wouldn't be opposed to some others moving off of Projects in a similar
manner. We can worry about the main osgeo services separately.
This of course leaves some questions:
Who can help setup the new base vm?
Should we implement puppet or chef, etc to make managing more servers
easier?
Should we consider using LXC containers to isolate projects from each
other on a common kernel?
I agree this is a good time to remind people that if they have specific
needs, such as things that adhoc is currently used for, then they need
to speak up.
The current defacto policy, we don't back it up unless you explicitly
asked for it to be backed up. We may also require assistance prepping
the files for backup if say a db backup is required. Obviously we are
backing some stuff up, but I'm not confident that we could really
restore a machine 100% from it.
As we're discovering, full backups of everything is unrealistic. It just
leads to dismal performance and overuse of the hardware. There are also
2 levels to consider:
1. Failover copy (we don't currently do this - mirrors is one
option,drbd another)
2. An actual archive backup used to build a fresh machine (this is the
current backup system)
Option 2 when needed does involve significant downtime to restore and
participation by the people who set up a machine to start. It seems what
more people really want is Option 1. In my mind that requires actual
planning in the next round of hardware or services.
Is this a good time to remind everyone of the
heavily-used-yet-not-supported AdhocVM? http://wiki.osgeo.org/wiki/AdhocVM Several projects rely on this VM
heavily, yet there are no backups etc. This has been reminded to me
recently by another community member. So, I'm nudging
-jeff
On 2014-05-09, 4:20 PM, Alex Mandel wrote:
We're currently stuck in a weird spot with trying to upgrade to Debian
7, a relic of the original VM disk layout that doesn't get along with GRUB2.
The fastest way to get there seems to be to do a new Base VM based on
debian 7 and clone it. This would be a good idea if we want to start
making smaller VMs to split apart some services into more isolation.
On the one hand its more VMs or containers to deal with, on the other it
might bring happy faces and stability to more things.
Our other option is some hackery to try and make current Vms upgradeable.
Currently the only requests I've heard for a new VM is from QGIS who
wants to remove their cruft from the existing VM by starting fresh
(since they moved all but 2 sites to another server elsewhere anyways).
I wouldn't be opposed to some others moving off of Projects in a similar
manner. We can worry about the main osgeo services separately.
This of course leaves some questions:
Who can help setup the new base vm?
Should we implement puppet or chef, etc to make managing more servers
easier?
Should we consider using LXC containers to isolate projects from each
other on a common kernel?
On Fri, May 09, 2014 at 03:13:09PM -0700, Alex Mandel wrote:
As we're discovering, full backups of everything is unrealistic. It just
leads to dismal performance and overuse of the hardware.
I disagree, I'd say you're shooting the messenger. Instead, full
backups *is* realistic, if the disk subsystem is done right - well, and
as long as sufficient backup space is available, of course. Currently
our VM's are sitting on inefficient disk subsystems, that's why backups
are having a huge impact.
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
On Fri, May 09, 2014 at 03:13:09PM -0700, Alex Mandel wrote:
As we're discovering, full backups of everything is unrealistic. It just
leads to dismal performance and overuse of the hardware.
I disagree, I'd say you're shooting the messenger. Instead, full
backups *is* realistic, if the disk subsystem is done right - well, and
as long as sufficient backup space is available, of course. Currently
our VM's are sitting on inefficient disk subsystems, that's why backups
are having a huge impact.
Cheers,
Martin.
Aside from switching from Raid 6 to Raid 5 what else would improve this?
My understanding is that Raid 6 should not be significantly slower for
read operations than Raid 5 but I have no real world data for that.
XFS, ZFS or BTFS instead of ext3? You mentioned lots of small files
being involved at some point. Is it an issue of not optimizing for that?
I'd love full backup if that seems feasible. Shifting the timing seems
to have cleared some of the congestion but clearly something else is
still going on. We have identified some increased none-sense traffic but
not much else has changed, other than the drives/hardware which should
all be back to optimum state.
In thinking about other hardware how does one avoid this issue in the
purchasing stage?
On Fri, May 09, 2014 at 04:12:52PM -0700, Alex Mandel wrote:
Aside from switching from Raid 6 to Raid 5 what else would improve this?
Foremost I'd like to make clear that at least my *primary* concern is
*not* to talk about technologies but instead it is to make a clear
distinction beween cause and effect. Thousands or maybe millions of
servers are being backed up while in service without getting the users
into trouble, but our VM's do get into trouble.
I'd like to avoid creating the impression that the backup is the
culprit. No, it's not, instead the disk system is, because .... and
then we can start talking about the details. If we confuse cause and
effect, then people are probably going to draw false conclusions over
and over and we hardly can stop it.
My understanding is that Raid 6 should not be significantly slower for
read operations than Raid 5 but I have no real world data for that.
I'd say the key to more I/O performance is eliminating the HW RAID
controller and using really fast (15k) drives. We're having a wild mix
of read and write operations of different sizes and according to my
experience the OS does better optimization these days than a HW
ontroller would do. Moreover there's fewer latency if we remove the
intermediate step - which the HW controller actually is.
I've done a couple of conversions from HW RAID to direct disk access
(sometimes still using the same host controller but for pass-through
disk access only) and I regret none of them.
XFS, ZFS or BTFS instead of ext3?
Ah, yes, please let's eliminate Ext3
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
On Sat, May 10, 2014 at 9:46 AM, Martin Spott <Martin.Spott@mgras.net> wrote:
On Fri, May 09, 2014 at 04:12:52PM -0700, Alex Mandel wrote:
Aside from switching from Raid 6 to Raid 5 what else would improve this?
...
XFS, ZFS or BTFS instead of ext3?
Ah, yes, please let's eliminate Ext3
To throw in some experience: we migrated our FEM PGIS cluster from
ext3 to XFS several years ago and did not regret this decision. For a
recent 96TB low cost extension based on small microserver boxes we
selected GlusterFS on top of XFS formatted disks. Works ok so far.
XFS is now the default FS for RHEL 7 (we use Scientific Linux).
I would hesitate to consider BTFS. ZFS sounds interesting but no
experience. So XFS might be the "natural" choice since it is widely
used for years.
On Sat, May 10, 2014 at 10:45:10AM +0200, Markus Neteler wrote:
To throw in some experience: we migrated our FEM PGIS cluster from
ext3 to XFS several years ago and did not regret this decision.
*Before* XFS was available on Linux, my workplace machine at home was
an Indigo2 MI. After I accidendially turned off the main switch I was
very surprised because a) the machine recovered rather quickly after
turning it on again, almost as fast as on a regular boot and b) nothing
was lost. Thus, as soon as the first 'official' XFS patch was
available for Linux 2.4.4 in 2001 (!), I've been using it regularly on
Linux as well and I never experienced any data loss due to filesystem
errors.
I admit, earlier versions of XFS for Linux had been a little bit
sensitive to corrupt block devices and I don't want to get into the
position to fix XFS inode tables manually (like I did with Ext2), but
overall I'd say it's the most stable, 'native' solution, especially for
"low maintenance" (TM) production servers running Linux.
I've been using ZFS on a couple of Sun Ultra-something machines running
Solaris (production database and file servers for hundreds of users)
with shared, redundant FC-storage and found it to be *very* convenient
and performant, but, as far as I understand, we're not going to see
native ZFS support in Linux. FreeBSD does ZFS and the interface and
tools very much resemble those on Solaris.
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
On Fri, May 09, 2014 at 12:20:42PM -0700, Alex Mandel wrote:
The fastest way to get there seems to be to do a new Base VM based on
debian 7 and clone it. This would be a good idea if we want to start
making smaller VMs to split apart some services into more isolation.
I don't see the net benefit of "more isolation", to me that's a
solution looking for a problem. The problem we need to solve (TM) is
insufficient disk I/O, but we're not going to solve it by isolating
different services into more VM's.
From a practical point of view the difference between a) 12 different
services running in 6 VM's (2 per VM) hammering on a slow I/O subsystem
compared to b) 12 services running in 2 VM's (6 per VM) hammering on
the same I/O subsystem (or even one VM for every service) is rather
esoteric. The bottleneck is still going to be disk I/O, because from a
physical block device perspective the task hasn't changed.
The solution to our problem is improving disk I/O performance, reducing
latency in our widely mixed use case and once we've arrived there, I'm
sure (almost) everybody will be happy - even without "more isolation".
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
On Sat, May 10, 2014 at 02:23:46PM +0200, I wrote:
[...] The bottleneck is still going to be disk I/O, because from a
physical block device perspective the task hasn't changed.
BTW, for the sake of completeness:
I perfectly agree with isolating the LDAP service into a separate VM
("Secure") for security as well as HA reasons. I agree with separating
the main web service into a separate VM ("Web") because it relies on
Drupal 5.x which, in turn, requires an insecure and unsupported version
of PHP. I agree with having one VM to be administered by everybody
until it's dead or hacked and can quickly get shut down without further
concern (I guess that's our "Webextra" or the "Adhoc" VM).
I'm sure there are more *organisational* reasons for isolating certain
services, but we should take care of being specific about the
respective reasons and not raise unrealistic expectations like
performance improvements by isolating a service into it's own VM on the
same host. The underlying storage hardware simply doesn't now it's
serving different VM's - the recommendation to use the "noop" scheduler
inside the KVM VM's is my witness
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
On Sat, May 10, 2014 at 2:23 PM, Martin Spott <Martin.Spott@mgras.net> wrote:
The bottleneck is still going to be disk I/O, because from a
physical block device perspective the task hasn't changed.
The solution to our problem is improving disk I/O performance, reducing
latency in our widely mixed use case and once we've arrived there, I'm
sure (almost) everybody will be happy - even without "more isolation".
Question: is there any chance that OSUOSL lets us temporarily move our
VMs off the current host machine to a different one (perhaps after
shrinking their size to a minimum) in order to be able to redo the
RAID stuff etc - then move back our VMs?
On Sat, May 10, 2014 at 2:23 PM, Martin Spott <Martin.Spott@mgras.net> wrote:
The bottleneck is still going to be disk I/O, because from a
physical block device perspective the task hasn't changed.
The solution to our problem is improving disk I/O performance, reducing
latency in our widely mixed use case and once we've arrived there, I'm
sure (almost) everybody will be happy - even without "more isolation".
Question: is there any chance that OSUOSL lets us temporarily move our
VMs off the current host machine to a different one (perhaps after
shrinking their size to a minimum) in order to be able to redo the
RAID stuff etc - then move back our VMs?
On Mon, May 12, 2014 at 8:14 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:
On 05/10/2014 02:37 PM, Markus Neteler wrote:
On Sat, May 10, 2014 at 2:23 PM, Martin Spott <Martin.Spott@mgras.net> wrote:
The bottleneck is still going to be disk I/O, because from a
physical block device perspective the task hasn't changed.
The solution to our problem is improving disk I/O performance, reducing
latency in our widely mixed use case and once we've arrived there, I'm
sure (almost) everybody will be happy - even without "more isolation".
Question: is there any chance that OSUOSL lets us temporarily move our
VMs off the current host machine to a different one (perhaps after
shrinking their size to a minimum) in order to be able to redo the
RAID stuff etc - then move back our VMs?
On Sat, Aug 23, 2014 at 04:02:49PM +0200, Markus Neteler wrote:
On Mon, May 12, 2014 at 8:14 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:
> On 05/10/2014 02:37 PM, Markus Neteler wrote:
>> Question: is there any chance that OSUOSL lets us temporarily move our
>> VMs off the current host machine to a different one (perhaps after
>> shrinking their size to a minimum) in order to be able to redo the
>> RAID stuff etc - then move back our VMs?
>>
>> Markus
>
> I will ask.
Did you obtain any answer?
Generally speaking we could even store LVM snapshots of our VM's on the
"backup" machine - but unfortunately we can't, because access to the VM
hosts is too limited.
We could even create full filesystem dumps of the affected VM's, store
then on the "backup" machine. Bu we still need OSL support in order to
redo the RAID and virtualization setup.
Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
