[SAC] Virtualbox on Osgeo6

On Thu, Sep 28, 2017 at 11:46:16PM +0200, Martin Spott wrote:

On Thu, Sep 28, 2017 at 02:21:07PM -0700, Alex M wrote:

> The thing about Docker, is you don't let anyone into a Docker, Docker's
> are immutable from a configuration viewpoint. The way you fix a Docker
> is update the script to deploy it and redeploy.

This is one style of using Docker, maybe the most popular one, but "there's
more than one way to do it" (like in Perl). Setting up a base system,
SSH'ing into it like you would into a VM and arranging things inside is not
that uncommon.

Just to confirm what Martin is saying.
Files (configuration/data) subject to change could be mounted from
either host or another ad-hoc data container.
Simplest would be a single docker with the code and data on the host.

--strk;

Hi Sandro,

yes its possible, basically we could add another property "Committee" to the OSGeo Members, and set it to "SAC" for all SAC members, or we could add the "Category::SAC" to the wiki user pages of each SAC member, and create a map from the according query.

But as said before, currently I do not have the resources to cope with the OSGeo wiki stuff ... I can not do it in the coming days/weeks, but I would be available and happy to answer any questions by mail on how to do it, if someone will take over caring for the wiki. It could also use an update of the underlying Mediawiki...

Best regards,
Christian

Am 27.09.2017 um 09:38 schrieb Sandro Santilli:

Christian: is the current wiki capable of showing "all SAC members on the map" ?

--strk;

Hi Christian,

I'll help by tackling the MediaWiki upgrade (https://trac.osgeo.org/osgeo/ticket/2004).

-jeff

On 2017-09-29 11:59 AM, Christian Willmes wrote:

Hi Sandro,

yes its possible, basically we could add another property "Committee" to the OSGeo Members, and set it to "SAC" for all SAC members, or we could add the "Category::SAC" to the wiki user pages of each SAC member, and create a map from the according query.

But as said before, currently I do not have the resources to cope with the OSGeo wiki stuff ... I can not do it in the coming days/weeks, but I would be available and happy to answer any questions by mail on how to do it, if someone will take over caring for the wiki. It could also use an update of the underlying Mediawiki...

Best regards,
Christian

Am 27.09.2017 um 09:38 schrieb Sandro Santilli:

Christian: is the current wiki capable of showing "all SAC members on the map" ?

--strk;

Hi Christian,

Christian Willmes wrote:

But as said before, currently I do not have the resources to cope with
the OSGeo wiki stuff ... I can not do it in the coming days/weeks, but I
would be available and happy to answer any questions by mail on how to
do it, if someone will take over caring for the wiki. It could also use
an update of the underlying Mediawiki...

Should we prepare for surprises concerning Wiki extensions during such
upgrade ?

  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

I hope not.

But of course better be prepared, than sorry later.

Backup of wiki web server folder and database should be fine.

- Christian

Am 15.10.2017 um 09:55 schrieb Martin Spott:

Hi Christian,

Christian Willmes wrote:

But as said before, currently I do not have the resources to cope with
the OSGeo wiki stuff ... I can not do it in the coming days/weeks, but I
would be available and happy to answer any questions by mail on how to
do it, if someone will take over caring for the wiki. It could also use
an update of the underlying Mediawiki...

Should we prepare for surprises concerning Wiki extensions during such
upgrade ?

  Martin.

Markus Neteler and I had a little chat this morning.

We were talking about how to isolate CMS'es like Wordpress in order to
reduce the potential risk that evolves from running your own. This still
doesn't reduce the risk of potential break-in's into the CMS itself, but at
least the rest of your environment remains unaffected.

As always, there are several options:

1.) Virtualize the entire host containing web server, CMS and database.
2.) Containerize the entire host containing web server, CMS and database.
3.) Containerize the web server and CMS into one instance and the database
    into a second.
4.) Containerize the web server and CMS and run the database on the host.
5.) Choose your favourite not listed here.

Now, there's a neat virtualization technique which could have saved us from
the hassle we're facing wrt. upgrading old VM's on both of our hosts (and
more), but it's not very popular in OSGeo land. Moreover, virtualization
always bears more overhead than containerization, so let's keep this can of
worms closed.

Second, let's look at the containerization techniques available today, with
rkt and Docker being among the most popular ones. They allow
containerization of just a small application environment, just to fit the
needs of a webserver, a database, whatever you like. As far as I can tell,
rkt and Docker could even co-exist on the same host.
Moreover there's LXC, a tool I like a lot because it has so little overhead
and which I consider perfect for running even a full system at much less
overhead than in full virtualization. They all rely on Linux Control
Groups.

Nowadays Docker is pretty bloated - but apparently "everybody" (TM) loves
it. Therefore, hoping that it will avoid heated discussions, I'm herewith
suggesting to containerize certain services into Docker. Note: I do *not*
suggest to let everybody containerize their garbage on OSGeo hosts in a
"fire and let others deal with the trouble they cause"-manner just because
there's a platform that would be able to run it. Careful selection should
still remain a core principle. I'm suggesting to dockerize the web server
and CMS parts only and keep the database instance(s) on the host, simply to
ease the database backup procedure and because I see little net benefit in
containerizing the DB as well.

A reverse proxy on the host would serve as a web gateway from outside, it
would even be able to terminate SSL encryption, if needed/wanted.

If we manage to reach consensus, then we'd start by dockerizing a new GRASS
web server instance on Osgeo6 to act as a guinea pig for the entire
procedure.

And while we're at it, I'd ask for permission to remove the remains of
VirtualBox from Osgeo6.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Not part of sac so this is just opinion but I would containerize the db as
well (the db files themselves would be outside of the container). It makes
upgrading of the database trivial and isolates all the components of the
db and its dependencies together.

Just my 2cents.

Mike

Michael Smith
OSGeo Foundation Treasurer
Treasurer@osgeo.org

-----Original Message-----
From: Sac <sac-bounces@lists.osgeo.org> on behalf of Martin Spott
<Martin.Spott@mgras.net>
Organization: home
Reply-To: System Administration Committee Discussion/OSGeo
<sac@lists.osgeo.org>
Date: Saturday, October 21, 2017 at 6:12 PM
To: <sac@lists.osgeo.org>
Subject: Re: [SAC] Virtualbox on Osgeo6

Markus Neteler and I had a little chat this morning.

We were talking about how to isolate CMS'es like Wordpress in order to
reduce the potential risk that evolves from running your own. This still
doesn't reduce the risk of potential break-in's into the CMS itself, but
at
least the rest of your environment remains unaffected.

As always, there are several options:

1.) Virtualize the entire host containing web server, CMS and database.
2.) Containerize the entire host containing web server, CMS and database.
3.) Containerize the web server and CMS into one instance and the database
   into a second.
4.) Containerize the web server and CMS and run the database on the host.
5.) Choose your favourite not listed here.

Now, there's a neat virtualization technique which could have saved us
from
the hassle we're facing wrt. upgrading old VM's on both of our hosts (and
more), but it's not very popular in OSGeo land. Moreover, virtualization
always bears more overhead than containerization, so let's keep this can
of
worms closed.

Second, let's look at the containerization techniques available today,
with
rkt and Docker being among the most popular ones. They allow
containerization of just a small application environment, just to fit the
needs of a webserver, a database, whatever you like. As far as I can
tell,
rkt and Docker could even co-exist on the same host.
Moreover there's LXC, a tool I like a lot because it has so little
overhead
and which I consider perfect for running even a full system at much less
overhead than in full virtualization. They all rely on Linux Control
Groups.

Nowadays Docker is pretty bloated - but apparently "everybody" (TM) loves
it. Therefore, hoping that it will avoid heated discussions, I'm herewith
suggesting to containerize certain services into Docker. Note: I do *not*
suggest to let everybody containerize their garbage on OSGeo hosts in a
"fire and let others deal with the trouble they cause"-manner just because
there's a platform that would be able to run it. Careful selection should
still remain a core principle. I'm suggesting to dockerize the web server
and CMS parts only and keep the database instance(s) on the host, simply
to
ease the database backup procedure and because I see little net benefit in
containerizing the DB as well.

A reverse proxy on the host would serve as a web gateway from outside, it
would even be able to terminate SSL encryption, if needed/wanted.

If we manage to reach consensus, then we'd start by dockerizing a new
GRASS
web server instance on Osgeo6 to act as a guinea pig for the entire
procedure.

And while we're at it, I'd ask for permission to remove the remains of
VirtualBox from Osgeo6.

Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac

On Sat, Oct 21, 2017 at 10:12:31PM +0000, Martin Spott wrote:

If we manage to reach consensus, then we'd start by dockerizing a new GRASS
web server instance on Osgeo6 to act as a guinea pig for the entire
procedure.

Does the GRASS web server need to be moved or is this excercise
just for a test ? I'm asking because there are other services
that need to be moved to free up space to host a new machine.

Here are the services that need to be moved:
https://wiki.osgeo.org/wiki/SAC_Service_Status#osgeo4
and in particular, the AdHoc VM services:
https://wiki.osgeo.org/wiki/AdhocVM
Alex please correct me if I'm wrong.

--strk;

On Oct 22, 2017 11:02 AM, “Sandro Santilli” <strk@kbt.io> wrote:

On Sat, Oct 21, 2017 at 10:12:31PM +0000, Martin Spott wrote:

If we manage to reach consensus, then we’d start by dockerizing a new GRASS
web server instance on Osgeo6 to act as a guinea pig for the entire
procedure.

Does the GRASS web server need to be moved or is this excercise
just for a test ?

Not a test but we want to migrate our outdated CMSMS to WP.
Note that all bigger files will not be replicated but just referred to. Hence the disk space consumption is expected to be minor since some screenshots and mainly text will define the new WP instance.

I’m asking because there are other services
that need to be moved to free up space to host a new machine.

The footprint of docker isn’t that much, right?

Here are the services that need to be moved:
https://wiki.osgeo.org/wiki/SAC_Service_Status#osgeo4
and in particular, the AdHoc VM services:
https://wiki.osgeo.org/wiki/AdhocVM
Alex please correct me if I’m wrong.

How much space will they require?

Markus

–strk;


Sac mailing list
Sac@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac

On Sun, Oct 22, 2017 at 11:30:11AM +0200, Markus Neteler wrote:

Not a test but we want to migrate our outdated CMSMS to WP.

Ok, makes sense to do on osgeo6 then.

> Here are the services that need to be moved:
> https://wiki.osgeo.org/wiki/SAC_Service_Status#osgeo4
> and in particular, the AdHoc VM services:
> https://wiki.osgeo.org/wiki/AdhocVM
> Alex please correct me if I'm wrong.

How much space will they require?

If everything is under /osgeo as it looks there are 32GB
of files in there. Plus there are 100M of PostgreSQL db
and 20M of MySQL files.

Maybe it would be easier to ask each of the services
administrators to dockerize the service for easier
moving it around ? I'm adding in Cc:

  - Astrid for the mapbender demo
  - Jeff for the mapserver demo
  - Tom for the pycsw and pywps demos and for the
    geohealtcheck service
  - Christopher for the OpenArealMap

Also there's an earthqake related cronjob implemented with
GRASS which you're listed as a contact of.

Could everyone please give their comment on the status
of the service they are listed as administrators of
and comment on their ability to help with migrating it
elsewhere ?

Status would be best updated on the specific wiki page:
https://wiki.osgeo.org/wiki/AdhocVM

Thank you.

--strk;

On Sun, Oct 22, 2017 at 12:17:26PM +0200, Sandro Santilli wrote:

I'm adding in Cc:

  - Astrid for the mapbender demo
  - Jeff for the mapserver demo
  - Tom for the pycsw and pywps demos and for the
    geohealtcheck service
  - Christopher for the OpenArealMap

FYI: the mail for Chris is outdated in the LDAP database so please
     take that address off the recipients list unless you have the
     correct one (I sent a separate mail about this).

--strk;

On Sun, Oct 22, 2017 at 12:17 PM, Sandro Santilli <strk@kbt.io> wrote:

Also there's an earthqake related cronjob implemented with
GRASS which you're listed as a contact of.

I checked and the cronjob is fully functional (i.e. earthquake map is
daily generated).
I removed unused stuff (25%), the remaining earthquake map job can now
moved as-is. Happy to help if needed.

Markus

On Sun, Oct 22, 2017 at 12:17 PM, Sandro Santilli <strk@kbt.io> wrote:

> Here are the services that need to be moved:
> https://wiki.osgeo.org/wiki/SAC_Service_Status#osgeo4
> and in particular, the AdHoc VM services:
> https://wiki.osgeo.org/wiki/AdhocVM
> Alex please correct me if I'm wrong.

If everything is under /osgeo as it looks there are 32GB
of files in there.

Perhaps some projects could try to reduce their disk consumption on
the AdHoc VM:

adhoc:/osgeo# du -hx . --max-depth=1 | sort -h # dusage
80K ./.svn
100K ./banjo_dev
61M ./oam
87M ./demo.pywps.org
90M ./mapmill
108M ./mapbender
132M ./demo.geohealthcheck.org
189M ./dev.geohealthcheck.org
231M ./demo.pycsw.org
1.2G ./livedvd
1.4G ./grass
28G ./mapserver

TOTAL: 31G

(I just reduced ./grass from 2GB to 1.4GB)

Plus there are 100M of PostgreSQL db
and 20M of MySQL files.

Markus

Martin Spott wrote:

Nowadays Docker is pretty bloated - but apparently "everybody" (TM) loves
it. Therefore, hoping that it will avoid heated discussions, I'm herewith
suggesting to containerize certain services into Docker.

[...]

And while we're at it, I'd ask for permission to remove the remains of
VirtualBox from Osgeo6.

May I proceed ?

  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Michael Smith wrote:

Not part of sac so this is just opinion but I would containerize the db as
well (the db files themselves would be outside of the container). It makes
upgrading of the database trivial and isolates all the components of the
db and its dependencies together.

I understand your point, but my view onto the topic is a different one.

I wonder if anybody here is happy to maintain and backup a crowd of MySQL
servers in individual containers or VM's. I personally don't, therefore I'd
be happy to aggregate the MySQL-requirements into a single or at least as
few as possible instance and do proper maintenance and backup just there.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

On Nov 8, 2017 10:51 AM, “Martin Spott” <Martin.Spott@mgras.net> wrote:

Martin Spott wrote:

Nowadays Docker is pretty bloated - but apparently “everybody” ™ loves
it. Therefore, hoping that it will avoid heated discussions, I’m herewith
suggesting to containerize certain services into Docker.
[…]
And while we’re at it, I’d ask for permission to remove the remains of
VirtualBox from Osgeo6.

May I proceed ?

Yes please, at least from my side.

We’ll have a GRASS GIS sprint over the next weekend and would like to start with WP in docker.

Thanks
Markus

Martin.

Unix IS user friendly - it’s just selective about who its friends are !


Sac mailing list
Sac@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac

On 11/08/2017 02:18 AM, Martin Spott wrote:

Michael Smith wrote:

Not part of sac so this is just opinion but I would containerize the db as
well (the db files themselves would be outside of the container). It makes
upgrading of the database trivial and isolates all the components of the
db and its dependencies together.

I understand your point, but my view onto the topic is a different one.

I wonder if anybody here is happy to maintain and backup a crowd of MySQL
servers in individual containers or VM's. I personally don't, therefore I'd
be happy to aggregate the MySQL-requirements into a single or at least as
few as possible instance and do proper maintenance and backup just there.

Cheers,
  Martin.

I think he meant 1 mysql container that all the other containers can
point to.

That said I find system packages for a db to be much easier, since we do
not run unusual variants of the db. This would be different if say
someone needed always needed the absolute latest release because some
extension required it. i.e. Postgres 10 with Postgis 2.4.x is probably
not packaged for stable debian as fast as some would want.

Thanks,
Alex

On 11/08/2017 03:48 AM, Markus Neteler wrote:

On Nov 8, 2017 10:51 AM, "Martin Spott" <Martin.Spott@mgras.net> wrote:

Martin Spott wrote:

Nowadays Docker is pretty bloated - but apparently "everybody" (TM)

loves

it. Therefore, hoping that it will avoid heated discussions, I'm

herewith

suggesting to containerize certain services into Docker.

[...]

And while we're at it, I'd ask for permission to remove the remains of
VirtualBox from Osgeo6.

May I proceed ?

Yes please, at least from my side.

We'll have a GRASS GIS sprint over the next weekend and would like to start
with WP in docker.

Thanks
Markus

Markus,

Great, if you make a reasonable Wordpress Docker, and figure out how to
keep it updated, we can reuse that base for the main OSGeo site as we
bring it over to our servers.

Thanks,
Alex

On Wed, Nov 08, 2017 at 09:51:35AM +0000, Martin Spott wrote:

Martin Spott wrote:

> Nowadays Docker is pretty bloated - but apparently "everybody" (TM) loves
> it. Therefore, hoping that it will avoid heated discussions, I'm herewith
> suggesting to containerize certain services into Docker.
[...]
> And while we're at it, I'd ask for permission to remove the remains of
> VirtualBox from Osgeo6.

May I proceed ?

Green light from me.
I'm happy to confirm NO virtualization is in use for Gogs/Drone :slight_smile:

--strk;

On 11/08/2017 11:51 AM, Sandro Santilli wrote:

On Wed, Nov 08, 2017 at 09:51:35AM +0000, Martin Spott wrote:

Martin Spott wrote:

Nowadays Docker is pretty bloated - but apparently "everybody" (TM) loves
it. Therefore, hoping that it will avoid heated discussions, I'm herewith
suggesting to containerize certain services into Docker.

[...]

And while we're at it, I'd ask for permission to remove the remains of
VirtualBox from Osgeo6.

May I proceed ?

Green light from me.
I'm happy to confirm NO virtualization is in use for Gogs/Drone :slight_smile:

--strk;
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac

Yes please clean the virtualization.

Thanks,
Alex