[SAC] download server: No space left on device

FYI

df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 142G 0 100% /

In order to contribute, I have deleted all RC tarballs of GRASS GIS 7.

df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 141G 398M 100% /

... not giving much effect, though.

TOP10 disk space consumers:

download:/osgeo/download# dusage.sh | tac | nl
     1 28458 osgeo4w
     2 22940 ossim
     3 22506 qgis
     4 17393 mapguide
     5 7696 webdav
     6 5824 fdo
     7 4882 postgis
     8 4812 logs
     9 3829 pgrouting
    10 3589 livedvd

At least the TOP4 please check how to reduce their disk consumption if
all possible.

Markus

Markus, Jurgen,

I see there are 10GB of stuff under /osgeo/download/osgeo4w/symstore
which appear to be Visual Studeio debug support files. I'm wondering
if they can be removed or trimmed.

Best regards,
Frank

On Sun, Oct 2, 2016 at 1:20 PM, Markus Neteler <neteler@osgeo.org> wrote:

FYI

df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 142G 0 100% /

In order to contribute, I have deleted all RC tarballs of GRASS GIS 7.

df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 141G 398M 100% /

... not giving much effect, though.

TOP10 disk space consumers:

download:/osgeo/download# dusage.sh | tac | nl
     1 28458 osgeo4w
     2 22940 ossim
     3 22506 qgis
     4 17393 mapguide
     5 7696 webdav
     6 5824 fdo
     7 4882 postgis
     8 4812 logs
     9 3829 pgrouting
    10 3589 livedvd

At least the TOP4 please check how to reduce their disk consumption if
all possible.

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

Hi Frank,

On Sun, 02. Oct 2016 at 14:28:29 -0700, Frank Warmerdam wrote:

Markus, Jurgen,

I see there are 10GB of stuff under /osgeo/download/osgeo4w/symstore
which appear to be Visual Studeio debug support files. I'm wondering
if they can be removed or trimmed.

Those are already constantly aged/cleaned (max. 7 days I think).

I cleaned out old installer in qgis instead. Now we have 8 GB again.

ossim seems to have a lot of files from 2009. Are those still relevant?

Or can we simply get more space?

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

Hi,

On Mon, 03. Oct 2016 at 01:30:37 +0200, Jürgen E. Fischer wrote:

> I see there are 10GB of stuff under /osgeo/download/osgeo4w/symstore
> which appear to be Visual Studeio debug support files. I'm wondering
> if they can be removed or trimmed.

As this happend again (and again and again...) - I moved the symstore to our
server. So there's 14GB room again now.

ossim seems to have a lot of files from 2009. Are those still relevant?

Are they?

Or can we simply get more space?

Can we?

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de
QGIS release manager (PSC) Germany IRC: jef on FreeNode

On Fri, Mar 24, 2017 at 09:03:40AM +0100, Jürgen E. Fischer wrote:

On Mon, 03. Oct 2016 at 01:30:37 +0200, Jürgen E. Fischer wrote:

> Or can we simply get more space?

Can we?

I guess the question at this point is: who's in a position to answer
that question ? I know there are economic resources to do it (check
SAC budget) so it must be just a matter of figuring out who would
do it and coordinating the communication with board or treasurer
for the actual expense involved.

Is anyone from SAC technically capable of getting more space into
that machine ?

--strk;

On 03/24/2017 12:30 PM, Sandro Santilli wrote:

On Fri, Mar 24, 2017 at 09:03:40AM +0100, Jürgen E. Fischer wrote:

On Mon, 03. Oct 2016 at 01:30:37 +0200, Jürgen E. Fischer wrote:

Or can we simply get more space?

Can we?

I guess the question at this point is: who's in a position to answer
that question ? I know there are economic resources to do it (check
SAC budget) so it must be just a matter of figuring out who would
do it and coordinating the communication with board or treasurer
for the actual expense involved.

Is anyone from SAC technically capable of getting more space into
that machine ?

--strk;

More space on the current server is not a financial issue. Technical,
only Martin and OSU have access to modify the ganeti disks. This is a
short term solution to expand or add another disk slice. The easiest way
is to mount a 2nd ganeti disk using some of the unallocated space from
the host. I can write up the commands if that helps, but don't have
access to do it myself. Growing the existing disk is also possible but
requires more than a reboot, and has a slight risk of data loss (as all
ext3 expansions do).

Longer term, I had put in the budget that we should replace osgeo3 with
a new machine this year. But only after osgeo4 is fully retired. So
really we just need to finish moving the last 2 VMs off osgeo4 so we can
clear up the rack space.

We could also just change where downloads is and put it on osgeo6, but
that isn't as cleanly separated from all the projects.

Thanks,
Alex

Hi,

On Fri, 24. Mar 2017 at 23:15:12 -0700, Alex Mandel wrote:

More space on the current server is not a financial issue. Technical,
only Martin and OSU have access to modify the ganeti disks. This is a
short term solution to expand or add another disk slice. The easiest way
is to mount a 2nd ganeti disk using some of the unallocated space from
the host. I can write up the commands if that helps, but don't have
access to do it myself. Growing the existing disk is also possible but
requires more than a reboot, and has a slight risk of data loss (as all
ext3 expansions do).

Could we please finally do something about this?

This is still an issue. I think it happened again yesterday/today. I have a
bunch of changed timestamps in osgeo4w now of unknown origin. I guess somebody
went through it and removed some (obsolete?) stuff.

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

On Thu, Oct 19, 2017 at 10:51:53AM +0200, Jürgen E. Fischer wrote:

Hi,

On Fri, 24. Mar 2017 at 23:15:12 -0700, Alex Mandel wrote:
> More space on the current server is not a financial issue. Technical,
> only Martin and OSU have access to modify the ganeti disks. This is a
> short term solution to expand or add another disk slice. The easiest way
> is to mount a 2nd ganeti disk using some of the unallocated space from
> the host. I can write up the commands if that helps, but don't have
> access to do it myself. Growing the existing disk is also possible but
> requires more than a reboot, and has a slight risk of data loss (as all
> ext3 expansions do).

Could we please finally do something about this?

This is still an issue. I think it happened again yesterday/today. I have a
bunch of changed timestamps in osgeo4w now of unknown origin. I guess somebody
went through it and removed some (obsolete?) stuff.

From http://wiki.osgeo.org/wiki/SAC_Service_Status I understand that

"download.osgeo.org" is a VM on a so-called "osgeo3" machine hosted
by OSUOSL, but nothing is found on that page about how to access it.
Neither osgeo3.osgeo.osuosl.org nor osgeo3.osuosl.org have a DNS
entry, so I don't even know the IP of that machine.

The same wiki page mentions that "Primary administrators" have ssh
keys to connect to physical machines. This set shuld include Alex and
Markus (and others) in addition to Martin, see:
https://wiki.osgeo.org/wiki/SAC:Primary_Administrators

Markus, Alex: do you not have acces to that machine ?
Martin: any reason not to share that Ganeti access with other SAC
members ? If we cannot control that machine maybe we should move
it elsewhere ?

Besides, the access file on "secure" contains some ganeti
related access credentials, but they don't seem to work.
Should we contact OSUOSL support ?

--strk;

On 10/19/2017 02:14 AM, Sandro Santilli wrote:

On Thu, Oct 19, 2017 at 10:51:53AM +0200, Jürgen E. Fischer wrote:

Hi,

On Fri, 24. Mar 2017 at 23:15:12 -0700, Alex Mandel wrote:

More space on the current server is not a financial issue. Technical,
only Martin and OSU have access to modify the ganeti disks. This is a
short term solution to expand or add another disk slice. The easiest way
is to mount a 2nd ganeti disk using some of the unallocated space from
the host. I can write up the commands if that helps, but don't have
access to do it myself. Growing the existing disk is also possible but
requires more than a reboot, and has a slight risk of data loss (as all
ext3 expansions do).

Could we please finally do something about this?

This is still an issue. I think it happened again yesterday/today. I have a
bunch of changed timestamps in osgeo4w now of unknown origin. I guess somebody
went through it and removed some (obsolete?) stuff.

From http://wiki.osgeo.org/wiki/SAC_Service_Status I understand that
"download.osgeo.org" is a VM on a so-called "osgeo3" machine hosted
by OSUOSL, but nothing is found on that page about how to access it.
Neither osgeo3.osgeo.osuosl.org nor osgeo3.osuosl.org have a DNS
entry, so I don't even know the IP of that machine.

The same wiki page mentions that "Primary administrators" have ssh
keys to connect to physical machines. This set shuld include Alex and
Markus (and others) in addition to Martin, see:
https://wiki.osgeo.org/wiki/SAC:Primary_Administrators

Markus, Alex: do you not have acces to that machine ?
Martin: any reason not to share that Ganeti access with other SAC
members ? If we cannot control that machine maybe we should move
it elsewhere ?

Besides, the access file on "secure" contains some ganeti
related access credentials, but they don't seem to work.
Should we contact OSUOSL support ?

--strk;

Martin is the only one who ever managed to get access to that host.
Partly because it required some hops via an osuosl backend service,
which is for security.

Adding a 2nd ganeti disk is probably the easiest option, resizing
existing disks is a hassle with real downtime (more than a restart).
Yes if you write up what you want we can put a ticket in to OSUOSL to
add the 2nd disk.

Here's the current disk allocation best I know
http://wiki.osgeo.org/wiki/Infrastructure_Transition_Plan_2010#osgeo3_.26_osgeo4

An NFS mount from osgeo6 is also an option. Could be done all by us.

-- Backstory --
OSUOSL offered to move us to their parallel FTP server system for
downloads, but something in the coordination never made that happen.
If someone wants to re-inquire about that, speak up.

We originally didn't have direct access because OSUOSL was managing a
custom gentoo install for the hosts. We've all learned years later that
we should have done it a different way. At this point osgeo3 should be
working towards retirement.

osgeo7, yet to be purchased is the planned replacement/upgrade of osgeo3
where we would configure it ourselves, like osgeo6.

However osgeo7 can't happen without retiring OSGeo4 (we need the rack
space and power(PDU space)), which means moving the QGIS and Adhoc VM
components to osgeo6. As I've noted QGIS materials could be moved at any
time. No idea what to do with Adhoc, containers would be nice, but maybe
a straight VM copy is easiest.

Thanks,
Alex

Hi Alex,

On Thu, 19. Oct 2017 at 09:47:36 -0700, Alex M wrote:

An NFS mount from osgeo6 is also an option. Could be done all by us.

Done. /dev/ogdata/download created on osgeo6 (100G), nfs exported, mounted on
download:/osgeo/download/download6, osgeo4w and qgis copied (42G) to it and
symlinked to the original spot.

For now qgis and osgeo4w were moved to a subdirectory named disabled - I'll
remove it if nothing else pops up for a while...

Not sure what we need to do on on backup - there's enough space to hold more
copies in the /mirror directory - and it already carries a lot of cruft that is
long gone from download.

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

On Fri, Oct 20, 2017 at 12:00:57AM +0200, Jürgen E. Fischer wrote:

Done. /dev/ogdata/download created on osgeo6 (100G)

[..]

Not sure what we need to do on on backup

Download backup strategy is documented to be based on rsync
to copy things to osgeo2, see:
https://wiki.osgeo.org/wiki/SAC:Backups#download.osgeo.org

Can you please also update https://wiki.osgeo.org/wiki/Download_Server
with the new state of NFS mounts ?

--strk;