[SAC] Upgrade list for VMs

Committees SAC mailing list
1

I'm trying to compile a list of modifications we want to make to VMs
that might require OSUOSL admin help:

1. Increase disk space of lists(aka Mail) server +10 GB
2. Increase disk space of qgis server +10 GB
3. Increase cpu allocation to qgis server to 3
4. Increase cpu allocation to trac server to 5
Anything else? I'd like to get the request in today.

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)
* Finish migrating osgeo1 to OSL servers (Drupal)
* Purchase of 3rd server for OSL - should it be slow with large disks,
or another VM machine to spread the load?
* Upgade Trac, we have a cloned server to test, if we're not going to do
it we should turn off that clone and recover the resources.
* Migrate Vmap0 to telescience

This is mostly a mental dump before I take off for a week of vacation
likely without internet.

Thanks,
Alex

2

On 12/13/2012 11:41 AM, Alex Mandel wrote:

I'm trying to compile a list of modifications we want to make to VMs
that might require OSUOSL admin help:

1. Increase disk space of lists(aka Mail) server +10 GB
2. Increase disk space of qgis server +10 GB
3. Increase cpu allocation to qgis server to 3
4. Increase cpu allocation to trac server to 5
Anything else? I'd like to get the request in today.

Any comments, suggestions, discussion before I put in this request?

Thanks,
Alex

3

Alex,

It looks reasonable to me.

Best regards,
Frank

···

On Fri, Dec 14, 2012 at 10:20 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:

On 12/13/2012 11:41 AM, Alex Mandel wrote:

I’m trying to compile a list of modifications we want to make to VMs
that might require OSUOSL admin help:

  1. Increase disk space of lists(aka Mail) server +10 GB
  2. Increase disk space of qgis server +10 GB
  3. Increase cpu allocation to qgis server to 3
  4. Increase cpu allocation to trac server to 5
    Anything else? I’d like to get the request in today.

Any comments, suggestions, discussion before I put in this request?

Thanks,
Alex

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

4

On Thu, Dec 13, 2012 at 11:41:35AM -0800, Alex Mandel wrote:

1. Increase disk space of lists(aka Mail) server +10 GB

I vaguely remember we agreed on adding 20 G, but I don't mind as long
as we can add another 10 G later after adding 10 G now.

* Finish migrating osgeo1 to OSL servers (Drupal)

As far as I remember, most of this is already done, including the MySQL
DB and "/var/www/html/". Anyhow, both of them might already be out of
date and I think the Apache config is missing.

Tschuess,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

5

On Thu, Dec 13, 2012 at 8:41 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:

I'm trying to compile a list of modifications we want to make to VMs
that might require OSUOSL admin help:

1. Increase disk space of lists(aka Mail) server +10 GB
2. Increase disk space of qgis server +10 GB
3. Increase cpu allocation to qgis server to 3
4. Increase cpu allocation to trac server to 5

While the stream of munin warnings has been significantly reduced
due to the update, I still get continuously warnings from trac.
Apparently we are still underpowered there? But moreover the
threshold setting is simply not updated yet since we do no longer
reach the maximum:

http://webextra.osgeo.osuosl.org/munin/osgeo.org/trac.osgeo.org-cpu.html

I'd suggest to set the threshold from currently 400 to something higher.

Markus

6

On 12/29/2012 01:24 PM, Markus Neteler wrote:

On Thu, Dec 13, 2012 at 8:41 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:

I'm trying to compile a list of modifications we want to make to VMs
that might require OSUOSL admin help:

1. Increase disk space of lists(aka Mail) server +10 GB
2. Increase disk space of qgis server +10 GB
3. Increase cpu allocation to qgis server to 3
4. Increase cpu allocation to trac server to 5

While the stream of munin warnings has been significantly reduced
due to the update, I still get continuously warnings from trac.
Apparently we are still underpowered there? But moreover the
threshold setting is simply not updated yet since we do no longer
reach the maximum:

http://webextra.osgeo.osuosl.org/munin/osgeo.org/trac.osgeo.org-cpu.html

I'd suggest to set the threshold from currently 400 to something higher.

Markus

I've attempted to set cpu.user to 450. We'll see if that quiets it down
a little. I'm not sure we're underpowered now, but you are right that
the threshold did not change to reflect an added cpu.

Thanks,
Alex

7

On 12/13/2012 11:41 AM, Alex Mandel wrote:

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)
* Finish migrating osgeo1 to OSL servers (Drupal)
* Purchase of 3rd server for OSL - should it be slow with large disks,
or another VM machine to spread the load?
* Upgade Trac, we have a cloned server to test, if we're not going to do
it we should turn off that clone and recover the resources.
* Migrate Vmap0 to telescience

This is mostly a mental dump before I take off for a week of vacation
likely without internet.

Thanks,
Alex

Slight update:
* We have a round-robin DNS with telescience for serving Vmap0 so we're
not taking all the hits anymore. - thanks Martin
* Progress is being made on Drupal migration from osgeo1, I've begun
discussions with my coworker who is an expert on the topic and can
probably solve this. I will check if spending funds might make this
happen faster.

* New Server
I'm starting to think that we might want to get a new machine that is
more VM capable, as trac seems to use as many cores as we can give it.
Going this route, we could re-balance the VM load and free up more disk
on osgeo4 for more backup space.
I've restarted the conversation with OSUOSL since they made an offer to
use a donation from us for a new machine (not sure that ever went through).
Other plan, is just to get something with big disks, but I'm not sure
we'd use that smartly over more cpu power for higher loads (likely to
have quite a bit of disk anyways since price has come down so much)

*Trac upgrade
1. We should move to mod_wsgi - I started looking at how complicated
this would be, should be possible in place, 1 trac site at a time. (Easy)
2. We should upgrade trac, this can happen before or after adding server
3. It's easy to clone the running server, test the upgrade and then swap
the 2 VMs ip addresses. (Harder)

Anyone have comments or other topics to add?

Thanks,
Alex

8

Alex,

Somehow I don't see the original message in this thread.

On Trac - I think it is suffering from robots requesting expensive
things (timelines over large ranges was one thing I saw). I'll look
into that a bit more today and see if I can tune the robots.txt.

On "other items" - I'm wondering if we would be able to run some
windows VMs at OSU OSL? I experimented a bit with using an Amazon
instance for windows builds, and it worked but it starts to get at
least a wee bit expensive to keen a reasonable fast and large instance
running constantly. The downside to this is that it would require
somewhat different administration approaches than the set of similar
linux VMs.

Best regards,
Frank

On Thu, Jan 17, 2013 at 6:36 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:

On 12/13/2012 11:41 AM, Alex Mandel wrote:

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)
* Finish migrating osgeo1 to OSL servers (Drupal)
* Purchase of 3rd server for OSL - should it be slow with large disks,
or another VM machine to spread the load?
* Upgade Trac, we have a cloned server to test, if we're not going to do
it we should turn off that clone and recover the resources.
* Migrate Vmap0 to telescience

This is mostly a mental dump before I take off for a week of vacation
likely without internet.

Thanks,
Alex

Slight update:
* We have a round-robin DNS with telescience for serving Vmap0 so we're
not taking all the hits anymore. - thanks Martin
* Progress is being made on Drupal migration from osgeo1, I've begun
discussions with my coworker who is an expert on the topic and can
probably solve this. I will check if spending funds might make this
happen faster.

* New Server
I'm starting to think that we might want to get a new machine that is
more VM capable, as trac seems to use as many cores as we can give it.
Going this route, we could re-balance the VM load and free up more disk
on osgeo4 for more backup space.
I've restarted the conversation with OSUOSL since they made an offer to
use a donation from us for a new machine (not sure that ever went through).
Other plan, is just to get something with big disks, but I'm not sure
we'd use that smartly over more cpu power for higher loads (likely to
have quite a bit of disk anyways since price has come down so much)

*Trac upgrade
1. We should move to mod_wsgi - I started looking at how complicated
this would be, should be possible in place, 1 trac site at a time. (Easy)
2. We should upgrade trac, this can happen before or after adding server
3. It's easy to clone the running server, test the upgrade and then swap
the 2 VMs ip addresses. (Harder)

Anyone have comments or other topics to add?

Thanks,
Alex
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

9

Frank,

What do we need the Windows Vms to do? Which versions of Windows?

I've actually been testing this concept for a while and am willing to
host a couple myself at Davis for the purposes of build platforms. I
have access to all versions of windows (licensed) and have a platform
almost identical to OSUOSL.

PS: I've already tested Win 7 and Win 2008 RC2 on ganeti, they aren't
fast but they work.

Thanks,
Alex

On 01/28/2013 10:36 AM, Frank Warmerdam wrote:

Alex,

Somehow I don't see the original message in this thread.

On Trac - I think it is suffering from robots requesting expensive
things (timelines over large ranges was one thing I saw). I'll look
into that a bit more today and see if I can tune the robots.txt.

On "other items" - I'm wondering if we would be able to run some
windows VMs at OSU OSL? I experimented a bit with using an Amazon
instance for windows builds, and it worked but it starts to get at
least a wee bit expensive to keen a reasonable fast and large instance
running constantly. The downside to this is that it would require
somewhat different administration approaches than the set of similar
linux VMs.

Best regards,
Frank

On Thu, Jan 17, 2013 at 6:36 PM, Alex Mandel <tech_dev@wildintellect.com> wrote:

On 12/13/2012 11:41 AM, Alex Mandel wrote:

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)
* Finish migrating osgeo1 to OSL servers (Drupal)
* Purchase of 3rd server for OSL - should it be slow with large disks,
or another VM machine to spread the load?
* Upgade Trac, we have a cloned server to test, if we're not going to do
it we should turn off that clone and recover the resources.
* Migrate Vmap0 to telescience

This is mostly a mental dump before I take off for a week of vacation
likely without internet.

Thanks,
Alex

Slight update:
* We have a round-robin DNS with telescience for serving Vmap0 so we're
not taking all the hits anymore. - thanks Martin
* Progress is being made on Drupal migration from osgeo1, I've begun
discussions with my coworker who is an expert on the topic and can
probably solve this. I will check if spending funds might make this
happen faster.

* New Server
I'm starting to think that we might want to get a new machine that is
more VM capable, as trac seems to use as many cores as we can give it.
Going this route, we could re-balance the VM load and free up more disk
on osgeo4 for more backup space.
I've restarted the conversation with OSUOSL since they made an offer to
use a donation from us for a new machine (not sure that ever went through).
Other plan, is just to get something with big disks, but I'm not sure
we'd use that smartly over more cpu power for higher loads (likely to
have quite a bit of disk anyways since price has come down so much)

*Trac upgrade
1. We should move to mod_wsgi - I started looking at how complicated
this would be, should be possible in place, 1 trac site at a time. (Easy)
2. We should upgrade trac, this can happen before or after adding server
3. It's easy to clone the running server, test the upgrade and then swap
the 2 VMs ip addresses. (Harder)

Anyone have comments or other topics to add?

Thanks,
Alex
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

10

On Mon, Jan 28, 2013 at 10:42 AM, Alex Mandel
<tech_dev@wildintellect.com> wrote:

Frank,

What do we need the Windows Vms to do? Which versions of Windows?

Alex,

My primary goal is a collaborative build environment for OSGeo4W
packages. If things go well, I can imagine offering it for projects
to prepare their own standalone builds and possible for windows based
automated testing.

I don't have a particular version of windows in mind.

I've actually been testing this concept for a while and am willing to
host a couple myself at Davis for the purposes of build platforms. I
have access to all versions of windows (licensed) and have a platform
almost identical to OSUOSL.

Sweet! How/when can we proceed?

PS: I've already tested Win 7 and Win 2008 RC2 on ganeti, they aren't
fast but they work.

Well, I'm not sure what you mean by "not fast", but I did fine the
Amazon "tiny" instances pretty slow for windows builds.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

11

I just need to figure out how to get people authenticated and finish
installing 4 new 12 core nodes so I have more room for more Vms. It will
probably take a couple of weeks, though I have a Win7 machine I made for
benchmarking that we can test on already.

Any idea how to extend OSGeo LDAP to windows domain login?

I actually attempted to build such a VM a few years back, to try to do
open source (mingw) compiling of everything. Got stuck on geos or gdal
and gave up. Are we going the Visual Studio Express route?

Thanks,
Alex

On 01/28/2013 10:47 AM, Frank Warmerdam wrote:

On Mon, Jan 28, 2013 at 10:42 AM, Alex Mandel
<tech_dev@wildintellect.com> wrote:

Frank,

What do we need the Windows Vms to do? Which versions of Windows?

Alex,

My primary goal is a collaborative build environment for OSGeo4W
packages. If things go well, I can imagine offering it for projects
to prepare their own standalone builds and possible for windows based
automated testing.

I don't have a particular version of windows in mind.

I've actually been testing this concept for a while and am willing to
host a couple myself at Davis for the purposes of build platforms. I
have access to all versions of windows (licensed) and have a platform
almost identical to OSUOSL.

Sweet! How/when can we proceed?

PS: I've already tested Win 7 and Win 2008 RC2 on ganeti, they aren't
fast but they work.

Well, I'm not sure what you mean by "not fast", but I did fine the
Amazon "tiny" instances pretty slow for windows builds.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

12

On Mon, Jan 28, 2013 at 10:59:44AM -0800, Alex Mandel wrote:

Any idea how to extend OSGeo LDAP to windows domain login?

If "domain logon" is sufficient, that'll be rather easy to achieve,
LDAP plus Samba 3 will do reliably. ADS, according to my knowledge, is
a lot more complicated. Aside from that, there's "pGina" which allows
to hijack Windows authentication and divert to LDAP.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

13

On 01/28/2013 11:23 AM, Martin Spott wrote:

On Mon, Jan 28, 2013 at 10:59:44AM -0800, Alex Mandel wrote:

Any idea how to extend OSGeo LDAP to windows domain login?

If "domain logon" is sufficient, that'll be rather easy to achieve,
LDAP plus Samba 3 will do reliably. ADS, according to my knowledge, is
a lot more complicated. Aside from that, there's "pGina" which allows
to hijack Windows authentication and divert to LDAP.

Cheers,
  Martin.

I believe Domain auth would suffice. What do we need to setup on the
OSGeo side for that - Samba 3? Should we get started on that on the
Secure machine?

Thanks,
Alex

14

On Mon, Jan 28, 2013 at 10:59 AM, Alex Mandel
<tech_dev@wildintellect.com> wrote:

I just need to figure out how to get people authenticated and finish
installing 4 new 12 core nodes so I have more room for more Vms. It will
probably take a couple of weeks, though I have a Win7 machine I made for
benchmarking that we can test on already.

Any idea how to extend OSGeo LDAP to windows domain login?

Alex,

I don't know about that. It would be nice, but I don't see it as critical.
I feel it would be sufficient to supply userid/password to folks as needed
"by hand".

I actually attempted to build such a VM a few years back, to try to do
open source (mingw) compiling of everything. Got stuck on geos or gdal
and gave up. Are we going the Visual Studio Express route?

I build GEOS and GDAL with visual studio express and that would be my
intention.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

15

On 12/13/12, Alex Mandel <tech_dev@wildintellect.com> wrote:
...

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)

How hard would it be to add at least lists.osgeo.org to the set of
munin monitored machines? A new disk full (or whatever) would affect
almost 20k of subscribers.

Thanks and sorry for bothering,
Markus

16

On Sun, Feb 10, 2013 at 05:43:35PM +0100, Markus Neteler wrote:

On 12/13/12, Alex Mandel <tech_dev@wildintellect.com> wrote:

> Other things that have come up which should be addressed:
> * Make sure all the machines are on Munin (missing - Secure, Mail, Web)

How hard would it be to add at least lists.osgeo.org to the set of
munin monitored machines?

"lists" ^= "mail"

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

17

On Sun, Feb 10, 2013 at 5:57 PM, Martin Spott <Martin.Spott@mgras.net> wrote:

On Sun, Feb 10, 2013 at 05:43:35PM +0100, Markus Neteler wrote:

On 12/13/12, Alex Mandel <tech_dev@wildintellect.com> wrote:

> Other things that have come up which should be addressed:
> * Make sure all the machines are on Munin (missing - Secure, Mail, Web)

How hard would it be to add at least lists.osgeo.org to the set of
munin monitored machines?

"lists" ^= "mail"

I know but it is not there yet:
http://webextra.osgeo.osuosl.org/munin/index.html

ciao
Markus

18

On 02/10/2013 11:12 AM, Markus Neteler wrote:

On Sun, Feb 10, 2013 at 5:57 PM, Martin Spott <Martin.Spott@mgras.net> wrote:

On Sun, Feb 10, 2013 at 05:43:35PM +0100, Markus Neteler wrote:

On 12/13/12, Alex Mandel <tech_dev@wildintellect.com> wrote:

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)

How hard would it be to add at least lists.osgeo.org to the set of
munin monitored machines?

"lists" ^= "mail"

I know but it is not there yet:
http://webextra.osgeo.osuosl.org/munin/index.html

ciao
Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

It's easy to do. We just need to install the munin client on Mail and
add 3 lines to the munin master on webextra. I'll try to do it today.

Thanks,
Alex

19

On 02/11/2013 04:08 PM, Alex Mandel wrote:

On 02/10/2013 11:12 AM, Markus Neteler wrote:

On Sun, Feb 10, 2013 at 5:57 PM, Martin Spott <Martin.Spott@mgras.net> wrote:

On Sun, Feb 10, 2013 at 05:43:35PM +0100, Markus Neteler wrote:

On 12/13/12, Alex Mandel <tech_dev@wildintellect.com> wrote:

Other things that have come up which should be addressed:
* Make sure all the machines are on Munin (missing - Secure, Mail, Web)

How hard would it be to add at least lists.osgeo.org to the set of
munin monitored machines?

"lists" ^= "mail"

I know but it is not there yet:
http://webextra.osgeo.osuosl.org/munin/index.html

ciao
Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

It's easy to do. We just need to install the munin client on Mail and
add 3 lines to the munin master on webextra. I'll try to do it today.

Thanks,
Alex
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Done,

Needed to run munin-node-configure on mail, add the allow rule for the
osuosl IP range, restart the node. Then edit the munin master on
webextra to pick it up.

Thanks,
Alex