[SAC] /var/www/qgisdata full on qgis

Committees SAC mailing list
1

Hi SAC,

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda3 61920724 56584640 2190692 97% /
tmpfs 3064896 0 3064896 0% /lib/init/rw
udev 3060044 124 3059920 1% /dev
tmpfs 3064896 0 3064896 0% /dev/shm
/dev/vda1 101105 23222 72662 25% /boot
/dev/vdb1 104805972 104471000 334972 100% /var/www/qgisdata

qgisdata is one of the download disks. We serve documentation, repo's
and installers from that one.

is it possible to resize the disk?
currently people miss downloads of 2.16 :frowning:

Regards,

Richard Duivenvoorde

2

To be clear we're talking about:
qgis.osgeo.osuosl.org virtual machine

Richard, can you provide an estimate of how much growing room QGIS will
need for the next year?

If I'm not mistaken some QGIS downloads also come from
download.osgeo.org? Is the difference that qgis VM also stores the full
archive of past releases?

Possible solutions:
1. Grow the disk on the existing osgeo4.
2. Migrate the hosting of such files to osgeo6 (note osgeo6 also has
space for 3 more disks on top of current available).
3. Grow the disk on download and move files.

I'm in favor of option 2 as osgeo4 is slated to be retired already.

Thanks,
Alex

On 07/12/2016 01:23 AM, Richard Duivenvoorde wrote:

Hi SAC,

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda3 61920724 56584640 2190692 97% /
tmpfs 3064896 0 3064896 0% /lib/init/rw
udev 3060044 124 3059920 1% /dev
tmpfs 3064896 0 3064896 0% /dev/shm
/dev/vda1 101105 23222 72662 25% /boot
/dev/vdb1 104805972 104471000 334972 100% /var/www/qgisdata

qgisdata is one of the download disks. We serve documentation, repo's
and installers from that one.

is it possible to resize the disk?
currently people miss downloads of 2.16 :frowning:

Regards,

Richard Duivenvoorde
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

3

On 12-07-16 18:22, Alex Mandel wrote:

To be clear we're talking about:
qgis.osgeo.osuosl.org virtual machine

Richard, can you provide an estimate of how much growing room QGIS will
need for the next year?

Hi Alex,

/var/www/qgisdata is a separate mount/disk if I understand correctly:

/var/www/qgisdata$ du -hs *
11G debian
6.0G debian-ltr
8.0G debian-nightly
4.1G debian-nightly-ltr
7.9G debian-nightly-release
15G downloads
547M QGIS-Documentation
1.9G QGIS-Documentation-1.8
5.8G QGIS-Documentation-2.0
5.9G QGIS-Documentation-2.2
6.6G QGIS-Documentation-2.6
7.0G QGIS-Documentation-2.8
2.4G QGIS-Website
0 QGIS-Website-test
11M redminedump
2.8G ubuntugis
2.2G ubuntugis-ltr
5.6G ubuntugis-nightly
849M ubuntugis-nightly-ltr
1.6G ubuntugis-nightly-release

we need the upcoming year at least 2 or 3 more rounds of Documentation
(so say 30 Gb), some downloads (say 15Gb too).
Not sure how fast the debian and ubuntu repo's will grow, but lets do 25
for these too? (@Juergen?).

So max 70Gb?

If I'm not mistaken some QGIS downloads also come from
download.osgeo.org? Is the difference that qgis VM also stores the full
archive of past releases?

If I'm correct the downloads stuff (NOT the debian/ubuntu/docs) is
mirrored between some machines (I think to download.osgeo.org and
another machine of Juergen) to spread the load of downloading installers
after a release.

But I do not think it the actual 'downloads' is the problem, I could try
to slim down the QGIS-Documentation output a little.

Possible solutions:
1. Grow the disk on the existing osgeo4.
2. Migrate the hosting of such files to osgeo6 (note osgeo6 also has
space for 3 more disks on top of current available).
3. Grow the disk on download and move files.

I'm in favor of option 2 as osgeo4 is slated to be retired already.

For me off course #1 is easiest (for us).

for #2: /me looking at Juergen: are the debian and ubuntugis files also
created on osgeo4? Or rsynced from qgis2?
If last, is it an option to move those to osgeo6?

#3 would mean a higher burden on the two servers which are serving 2.16
now (though we already miss qgis.osgeo.osuosl.org now because rsyncing
stalled...)..

About retiring osgeo4: we are still investigating a move from redmine
(issues and wiki), IF that succeeds, we will only serve static files
from osgeo4 if I am correct...

Regards,

Richard Duivenvoorde

4

Hi Richard,

On Tue, 12. Jul 2016 at 10:23:03 +0200, Richard Duivenvoorde wrote:

qgisdata is one of the download disks. We serve documentation, repo's and
installers from that one.

is it possible to resize the disk?
currently people miss downloads of 2.16 :frowning:

Because the redirects were not updated. The (current) standalones are on
download.o.o (as is osgeo4w). The debian packages are served by qgis.org - but
those are in place (actually there were a lot of old packages as packages with
the version name in them don't get overwritten by new versions). There's now
~6GB free space:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vdb1 104805972 98619732 6186240 95% /var/www/qgisdata

The situation on download.o.o is similar:

Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 134G 7.2G 95% /

Why do we have to be so restrictive about disk space? On qgis2 (our 49€/month
machine at Hetzner with 2x2TB mirrored disks including 30TB of traffic) there
is still 1.24 TB of unallocated space. It feels like I've spend "years" of hetzner
server rent on making space or cleaning up and recovering from events caused
by running out of diskspace on osgeo machines (and maybe also caused that for
others on download.o.o).

Sure free space automatically fills up with cruft, but still 150GB nowadays
isn't that much...

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de
QGIS release manager (PSC) Germany IRC: jef on FreeNode

5

On Wed, Jul 13, 2016 at 08:59:58AM +0200, Richard Duivenvoorde wrote:

/var/www/qgisdata$ du -hs *
8.0G debian-nightly
4.1G debian-nightly-ltr
7.9G debian-nightly-release
5.6G ubuntugis-nightly
849M ubuntugis-nightly-ltr
1.6G ubuntugis-nightly-release

Just an idea: how many old nightly versions are you keeping ?
For its nature, does it make sense to keep more than a few
versions of nightlies ?

--strk;

6

On 07/13/2016 01:35 AM, Jürgen E. Fischer wrote:

Hi Richard,

On Tue, 12. Jul 2016 at 10:23:03 +0200, Richard Duivenvoorde wrote:

qgisdata is one of the download disks. We serve documentation, repo's and
installers from that one.

is it possible to resize the disk?
currently people miss downloads of 2.16 :frowning:

Because the redirects were not updated. The (current) standalones are on
download.o.o (as is osgeo4w). The debian packages are served by qgis.org - but
those are in place (actually there were a lot of old packages as packages with
the version name in them don't get overwritten by new versions). There's now
~6GB free space:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vdb1 104805972 98619732 6186240 95% /var/www/qgisdata

The situation on download.o.o is similar:

Filesystem Size Used Avail Use% Mounted on
/dev/vda3 149G 134G 7.2G 95% /

Why do we have to be so restrictive about disk space? On qgis2 (our 49€/month
machine at Hetzner with 2x2TB mirrored disks including 30TB of traffic) there
is still 1.24 TB of unallocated space. It feels like I've spend "years" of hetzner
server rent on making space or cleaning up and recovering from events caused
by running out of diskspace on osgeo machines (and maybe also caused that for
others on download.o.o).

Sure free space automatically fills up with cruft, but still 150GB nowadays
isn't that much...

Jürgen

We don't have to be so restrictive, osgeo4 has space but is a little bit
of work to grow.

We have much more space on osgeo6, and room to grow there, it's also SSD
drives, so faster. I think we bought 3x480GB in RAID5 if I recall, and 3
empty slots. Maybe we should fill those other slots this year. That's
why I would like to move the qgis downloads/docs there.

Even osgeo3 with download we'd been looking to grow, the project was
never finalized (osuosl was going to give us access to their big
http/ftp system). If someone has time to finish working that out, I'm
happy to pass that job to them.

Thanks,
Alex

7

On 07/13/2016 07:23 AM, Alex Mandel wrote:

We have much more space on osgeo6, and room to grow there, it's also SSD
drives, so faster. I think we bought 3x480GB in RAID5 if I recall, and 3
empty slots. Maybe we should fill those other slots this year. That's
why I would like to move the qgis downloads/docs there.

Correction, I looked up and updated the wiki page on osgeo6 hardware.
https://wiki.osgeo.org/wiki/Osgeo6

It's
2x80GB for the main drive (not sure the raid configure)
and
4x480GB RAID5 for the data drive, so 1.4 TB of space available just for
data.
All SSDs.

If it works for the QGIS team I'd like to move all the static web files
over to osgeo6. We can leave the redmine as is for now, with the goal of
retiring it later this year.

Thanks,
Alex

8

On 07/13/2016 07:39 AM, Alex Mandel wrote:

On 07/13/2016 07:23 AM, Alex Mandel wrote:

We have much more space on osgeo6, and room to grow there, it's also SSD
drives, so faster. I think we bought 3x480GB in RAID5 if I recall, and 3
empty slots. Maybe we should fill those other slots this year. That's
why I would like to move the qgis downloads/docs there.

Correction, I looked up and updated the wiki page on osgeo6 hardware.
https://wiki.osgeo.org/wiki/Osgeo6

It's
2x80GB for the main drive (not sure the raid configure)
and
4x480GB RAID5 for the data drive, so 1.4 TB of space available just for
data.
All SSDs.

If it works for the QGIS team I'd like to move all the static web files
over to osgeo6. We can leave the redmine as is for now, with the goal of
retiring it later this year.

Thanks,
Alex

Martin,

What will it take to setup Jurgen and Richard the ability to migrate the
static components of QGIS.org to osgeo6?

Are those lvm volumes on osgeo6? Should we make one for QGIS specifically?

Thanks,
Alex