[SAC] Adjust cron jobs on osgeo1?

Committees SAC mailing list
1

I'm wondering if adjusting some of the cron jobs on osgeo1 to other
times could alleviate the nightly/daily issue starting around GMT 8.

Can someone try adjusting a few jobs to other times?

European users are getting quite frustrated on IRC over the issue.

The only other solution I see is to move SVN and LDAP very soon to osl
VMs, but I don't think we have the people time to effectively do that
quickly right now.

Thanks,
Alex

PS: I'm willing to take a look at analyzing the cron jobs and system
logs if someone wants to give me access to osgeo1.

2

On May 26, 2010, at 5:18 AM, ext Alex Mandel wrote:

The only other solution I see is to move SVN and LDAP very soon to osl
VMs, but I don't think we have the people time to effectively do that
quickly right now.

I'm going to help with this approach. Moving cronjobs around seems unlikely to have a significant impact -- even with no obvious cronjobs running, performance suffers.

SVN and LDAP can be moved separately from each other, right? I assume that we want to have a separate VM for SVN, which presumably will either need to be the same VM as trac, or have a shared disk with trac. Have we specced that or set it up?

We already have a process in place for importing SVN exports from other servers. Following that same process for our existing SVN repos doesn't seem that difficult.

Assuming that we don't have a good way to have a shared disk, I would suggest that a VM for trac/SVN should have:

4GB RAM
500GB disk

I don't know our current distribution of resources, but trac/SVN are our most widely used services, so setting them up to have a significant chunk of resources seems wise. This would effectively give them double the capacity of OSGeo1, which seems likely to be sufficient at this time.

Once we get this VM set up, I will get the necessary bits installed. If we can get this done before next week, I can make time to do this next week. (If we can get it before the weekend, I can try to start work this weekend.)

Regards,
--
Christopher Schmidt
Nokia

3

christopher.schmidt@nokia.com wrote:

Assuming that we don't have a good way to have a shared disk, I would
suggest that a VM for trac/SVN should have:

4GB RAM 500GB disk

Chris,

I would note that SVN currently requires 20GB (/var/www/svn) and
Trac currently requires 700MB (/var/www/trac). I believe the large
server only has about 1000GB, and the smaller one even less so allocated
25 times the required space for the sake of growing room seems like it
will crowd out most other services. Perhaps we could allocate 120GB or
so?

I should have some availability to help with the transition of trac/svn
if I can be helpful. Don't hesitate to phone me at +1 613 628 2337 if
I'm not online.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

4

On May 26, 2010, at 9:14 AM, Frank Warmerdam wrote:

christopher.schmidt@nokia.com wrote:

Assuming that we don't have a good way to have a shared disk, I would
suggest that a VM for trac/SVN should have:
4GB RAM 500GB disk

Chris,

I would note that SVN currently requires 20GB (/var/www/svn) and
Trac currently requires 700MB (/var/www/trac). I believe the large
server only has about 1000GB, and the smaller one even less so allocated
25 times the required space for the sake of growing room seems like it
will crowd out most other services. Perhaps we could allocate 120GB or
so?

I should have some availability to help with the transition of trac/svn
if I can be helpful. Don't hesitate to phone me at +1 613 628 2337 if
I'm not online.

Same here. As long as the time is scheduled somewhat in advance, I can help with the transition as well.

5

christopher.schmidt@nokia.com wrote:
  > SVN and LDAP can be moved separately from each other, right? I assume that

we want to have a separate VM for SVN, which presumably will either need to
be the same VM as trac, or have a shared disk with trac. Have we specced
that or set it up?

Chris,

Just to confirm the plan is to have one VM for Trac and SVN. I see in
the table it was listed with 100-200GB and 8GB of RAM. We have quite a
bit of RAM so if 8GB might be helpful for performance, I think it is worth
allocating to this task.

I also wanted to note that our current Trac has custom hacks to use the
OSGeo LDAP, which I did. It might be possible to use built-in LDAP
support in the future, or I may need to reintroduce those changes in
whatever newer Trac we deploy. I am happy to help with this particular
part.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

6

On May 26, 2010, at 10:14 AM, ext Frank Warmerdam wrote:

christopher.schmidt@nokia.com wrote:

Assuming that we don't have a good way to have a shared disk, I would
suggest that a VM for trac/SVN should have:

4GB RAM 500GB disk

Chris,

I would note that SVN currently requires 20GB (/var/www/svn) and
Trac currently requires 700MB (/var/www/trac). I believe the large
server only has about 1000GB, and the smaller one even less so allocated
25 times the required space for the sake of growing room seems like it
will crowd out most other services. Perhaps we could allocate 120GB or
so?

Fair enough; somehow I had thought SVN was much bigger than that. I'd like to establish with projects that we recommend putting anything larger than 100MB not-in-SVN in that case (probably on Download instead); in that case 100GB seems sufficient to give us growing room.

-- Chris

I should have some availability to help with the transition of trac/svn
if I can be helpful. Don't hesitate to phone me at +1 613 628 2337 if
I'm not online.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Programmer for Rent

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
Christopher Schmidt
Nokia

7

On May 26, 2010, at 10:19 AM, ext Frank Warmerdam wrote:

christopher.schmidt@nokia.com wrote:

SVN and LDAP can be moved separately from each other, right? I assume that
we want to have a separate VM for SVN, which presumably will either need to
be the same VM as trac, or have a shared disk with trac. Have we specced
that or set it up?

Chris,

Just to confirm the plan is to have one VM for Trac and SVN. I see in
the table it was listed with 100-200GB and 8GB of RAM. We have quite a
bit of RAM so if 8GB might be helpful for performance, I think it is worth
allocating to this task.

Gotcha. I don't think that there is going to be a lot of difference between 4GB and 8GB; with our current setup, we're not even pushing 2GB most of the time (since I adjusted Apache children ages ago) and seperating out Drupal/PHP from the webserver serving Trac/SVN will leave us even less memory-per-apache child. I'm happy to go with either 4GB or 8GB and 100GB of disk, especially since we can re-allocate later.

I also wanted to note that our current Trac has custom hacks to use the
OSGeo LDAP, which I did. It might be possible to use built-in LDAP
support in the future, or I may need to reintroduce those changes in
whatever newer Trac we deploy. I am happy to help with this particular
part.

Currently, for trac LDAP integration, we just use LDAP as an authentication (verifying who a user is), correct? If that's the case, it looks the http://trac-hacks.org/wiki/LdapPlugin can solve that need. Once we get a VM set up, I'll experiment to see if I can confirm that.

I've edited the wiki page to set the disk to 100GB, and left the RAM at 8GB, since we're still vastly under the limit for RAM. Currently, the number of CPUs is listed as 2. In general, we have anywhere from 6-8 active SVN requests, so having more CPU would probably not hurt, but I don't understand how CPUs work in this environment; I assume that we don't have to match the total 'CPUs used' on the system to the number of actual cores, and can exceed that. If someone can confirm that, I'd bump that number of virtual CPUs to 4 instead of 2.

Regards,
--
Christopher Schmidt
Nokia

8

I've edited the wiki page to set the disk to 100GB, and left the RAM at 8GB, since we're still vastly under the limit for RAM. Currently, the number of CPUs is listed as 2. In general, we have anywhere from 6-8 active SVN requests, so having more CPU would probably not hurt, but I don't understand how CPUs work in this environment; I assume that we don't have to match the total 'CPUs used' on the system to the number of actual cores, and can exceed that. If someone can confirm that, I'd bump that number of virtual CPUs to 4 instead of 2.

After doing a bit of research, it looks like this assumption is correct. I'm upping the number of virtual CPUs to 4 on the Trac/SVN VM in the wiki.

So, our VM plan is:

* 100GB of disk (~20GB for SVN, 15GB for system, leaves with a factor-of-3-growth space for SVN)
* 8GB RAM (ram is cheap in this case)
* 4 CPUs

If anyone has an objection to this plan for the Trac/SVN VM, or other comments to make, please do so in the next 24 hours. (If you need more time to formulate a realistic response, just say so.) If not, then I hope Alex (or someone else?) can work with OSUOSL to set up the new VM tomorrow, so that work can begin on the transition this weekend.

Regards,
--
Christopher Schmidt
Nokia

9

christopher.schmidt@nokia.com wrote:

On May 26, 2010, at 5:18 AM, ext Alex Mandel wrote:

The only other solution I see is to move SVN and LDAP very soon to osl
VMs, but I don't think we have the people time to effectively do that
quickly right now.

I'm going to help with this approach. Moving cronjobs around seems unlikely to have a significant impact -- even with no obvious cronjobs running, performance suffers.

SVN and LDAP can be moved separately from each other, right? I assume that we want to have a separate VM for SVN, which presumably will either need to be the same VM as trac, or have a shared disk with trac. Have we specced that or set it up?

We already have a process in place for importing SVN exports from other servers. Following that same process for our existing SVN repos doesn't seem that difficult.

Assuming that we don't have a good way to have a shared disk, I would suggest that a VM for trac/SVN should have:

4GB RAM
500GB disk

I don't know our current distribution of resources, but trac/SVN are our most widely used services, so setting them up to have a significant chunk of resources seems wise. This would effectively give them double the capacity of OSGeo1, which seems likely to be sufficient at this time.

Once we get this VM set up, I will get the necessary bits installed. If we can get this done before next week, I can make time to do this next week. (If we can get it before the weekend, I can try to start work this weekend.)

Regards,

Yes Tracsvn is one VM and Secure is another.

FYI, Secure the new home for LDAP has been up since round 1, but there's
been little discussion lately about when to move it.

I'm going to put in the request for Tracsvn, and check if our backend
access to the hosts is ready for us to do it ourselves/adjust diskspace
& ram after creation.

Thanks,
Alex

10

christopher.schmidt@nokia.com wrote:

I've edited the wiki page to set the disk to 100GB, and left the RAM at 8GB, since we're still vastly under the limit for RAM. Currently, the number of CPUs is listed as 2. In general, we have anywhere from 6-8 active SVN requests, so having more CPU would probably not hurt, but I don't understand how CPUs work in this environment; I assume that we don't have to match the total 'CPUs used' on the system to the number of actual cores, and can exceed that. If someone can confirm that, I'd bump that number of virtual CPUs to 4 instead of 2.

After doing a bit of research, it looks like this assumption is correct. I'm upping the number of virtual CPUs to 4 on the Trac/SVN VM in the wiki.

So, our VM plan is:

* 100GB of disk (~20GB for SVN, 15GB for system, leaves with a factor-of-3-growth space for SVN)
* 8GB RAM (ram is cheap in this case)
* 4 CPUs

If anyone has an objection to this plan for the Trac/SVN VM, or other comments to make, please do so in the next 24 hours. (If you need more time to formulate a realistic response, just say so.) If not, then I hope Alex (or someone else?) can work with OSUOSL to set up the new VM tomorrow, so that work can begin on the transition this weekend.

Regards,

As some may have heard, this VM is up and ready. Are we on for migrating
Trac/SVN this weekend. Some users in the community have asked that we
send out a notice of potential downtime, or the time window in which
commits may be lost and should not be attempted.

Thanks,
Alex

11

On Fri, May 28, 2010 at 9:15 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:
...

As some may have heard, this VM is up and ready. Are we on for migrating
Trac/SVN this weekend. Some users in the community have asked that we
send out a notice of potential downtime, or the time window in which
commits may be lost and should not be attempted.

please do that - it is absolutely important (ideally 1-2 days in advance).

thanks & good luck,
Markus