[SAC] spatialref.org

Alex_M · June 10, 2013, 10:49pm

On 06/10/2013 03:18 PM, Alex Mandel wrote:

On 06/10/2013 03:14 PM, Howard Butler wrote:

On Jun 10, 2013, at 5:11 PM, "Hare, Trent" <thare@usgs.gov> wrote:

Guys,
Sorry if this is not the correct place to send this email. This morning http://spatialreference.org/ was working normally and now it appears the website has been replaced with a FOSS4G 2013.

FYI, I do have some IAU2000 updates and was planning to send them to Frank. Is this still correct?

The site got trashed by someone updating django this afternoon. I made a quick attempt to get it back online, but I don't understand how things are deployed anymore.

Howard

I'll get to it in a few minutes, actually have work to do right now.

This page is in fact up to date, though I should move the info to the
Trac site it seems.
http://wiki.osgeo.org/wiki/Spatialreference.org

Basics: sr.org runs on gunicorn on port 8092
Apache is just a proxy in front to map request on port 80 to 8092 and back.

Thanks,
Alex

Ok did some investigating, it's not a simple matter of django got
upgraded. It's actually based on the fact that python got upgraded
(would have broke a virtualenv too)

Easiest way to test is to bypass apache
http://spatialreference.org:8092/

Looking in /var/log/gunicorn/sr.gunicorn.log

File "/osgeo/sr.org/htdocs/srsbrowser/feeds.py", line 1, in <module>
from django.contrib.syndication.feeds import Feed
ImportError: No module named feeds

That seems to be renamed to
from django.contrib.syndication.views import Feed

Next error
File "/osgeo/sr.org/htdocs/urls.py", line 10, in <module>
(r'^admin/(.*)', admin.site.root),
AttributeError: 'AdminSite' object has no attribute 'root'

Fix, change line to (r'^admin/',include(admin.site.urls)),

Next Error
File "/osgeo/sr.org/htdocs/geocoder/views.py", line 7, in <module>
from geopy import geocoders
ImportError: No module named geopy

Fix, upgrade brought python 2.6, pip install geopy in python 2.6

Next Error
ImproperlyConfigured: Error importing template source loader
django.template.loaders.filesystem.load_template_source: "'module'
object has no attribute 'load_template_source'"

Fix in settings.py
(http://stackoverflow.com/questions/11904609/configuring-django-settings-to-work-with-1-4-1-loading-template-error)
'django.template.loaders.filesystem.load_template_source',
'django.template.loaders.app_directories.load_template_source',

Becomes:
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader

That at least get's it loading a page instead of a 500 error. Moving to
check apache now...

Thanks,
Alex

Arnulf_Christl4 · June 10, 2013, 11:05pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

<irrelevant rant>
Ahrg: HATE:

'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader

Why the BLAH can we not call a file system a filesystem and a software
a software! Whats wrong with a snippet of fucking software residing in
a fucking filesystem that you have to call it an "App"!
</irrelevant rant>

PS:
Maybe one or two glasses of wine have inspired this uncontrolled spill
of ancestral angular rigidity. Apologies.

- --
Seven of Nine
http://arnulf.us/Seven
Exploring Body, Speech and Mind
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlG2W6YACgkQXmFKW+BJ1b0EBgCeM7Lr4qVKopbgAxu4wFHyxzIE
u8oAnRRK2uBt/6WSgpHk9W1Y+s0d3HL+
=ABKI
-----END PGP SIGNATURE-----

Martin_Spott · June 11, 2013, 5:03am

On Mon, Jun 10, 2013 at 03:49:45PM -0700, Alex Mandel wrote:

Ok did some investigating, it's not a simple matter of django got
upgraded. It's actually based on the fact that python got upgraded
(would have broke a virtualenv too)

Indeed it's a different Django, no Python version change occurred. If
it causes too much hassle, then I'd simply revert to the previous
Django.

Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Alex_M · June 11, 2013, 5:53am

On 06/10/2013 10:03 PM, Martin Spott wrote:

On Mon, Jun 10, 2013 at 03:49:45PM -0700, Alex Mandel wrote:

Ok did some investigating, it's not a simple matter of django
got upgraded. It's actually based on the fact that python got
upgraded (would have broke a virtualenv too)

Indeed it's a different Django, no Python version change occurred.
If it causes too much hassle, then I'd simply revert to the
previous Django.

Cheers, Martin.

It's fixed now (Thanks for the help Chris), just needed the Database
section changed from:
DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = 'srorg'
DATABASE_USER = 'postgres'
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = '5432'

TO:

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis',
        'NAME': 'srorg',
        'USER': 'postgres',
        'PASSWORD': '',
        'HOST': '',
        'PORT': '5432',
    }
}

I will edit and commit the changes to the svn repo sr.org folder under
the MetaCRS trac.

Thanks,
Alex

Barry_Rowlingson · June 11, 2013, 7:01am

On Tue, Jun 11, 2013 at 12:05 AM, Seven (aka Arnulf) <seven@arnulf.us> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

<irrelevant rant>
Ahrg: HATE:

'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader

Why the BLAH can we not call a file system a filesystem and a software
a software! Whats wrong with a snippet of fucking software residing in
a fucking filesystem that you have to call it an "App"!
</irrelevant rant>

PS:
Maybe one or two glasses of wine have inspired this uncontrolled spill
of ancestral angular rigidity. Apologies.

The django filesystem loader is a python module that looks for
templates in a list of locations specified in a config file. The
app_directories loader looks for templates in a folder called
'templates' in all your django apps. It all makes sense really!

In django terms, an 'app' is a well-defined sub-section of a
'project'. The project is basically one web server and one database
per project, and there may be a blogging 'app', a CMS app, a FOSS4G
workshop booking app all sitting in that project.

The django app does need its own name, its a shame that marketing is
branding everything as 'apps' these days.

Would you be less irritated if they'd been called 'djanglets'?

Enjoy your hangover!

Barry

hobu · June 11, 2013, 4:00pm

... notes on spatialreference.org breaking due to system upgrades on the SAC Projects VM ...

I regret that I might have had a tone that implied whomever made changes to the project machine's django/python was to blame for things breaking. The reality is that spatialreference.org is a nearly six year old django application that has not been updated in more than three years, and it has simply atrophied in the face of the continuing march of progress. In short, it badly needs an update. Unfortunately, sr.org is one of those things that people love to use, but attracts no developer attention to love. Maybe it is time to decommission it with a note stating that it needs developer resources to bring it forward.

Howard

Basques_Bob_CI-StPau · June 11, 2013, 4:04pm

Howard,

Who owns spatialref.org?

I use it quite a bit myself, but it usually is the sort of thing where I need it now!! for a very tight timeline sort of thing. I often wondered about integrating it into something as a helper service of some sort, as well as extending the service a bit.

Bobb

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Howard Butler
Sent: Tuesday, June 11, 2013 11:01 AM
To: System Administration Committee Discussion/OSGeo
Cc: metacrs@lists.osgeo.org
Subject: Re: [SAC] spatialref.org

... notes on spatialreference.org breaking due to system upgrades on the SAC Projects VM ...

I regret that I might have had a tone that implied whomever made changes to the project machine's django/python was to blame for things breaking. The reality is that spatialreference.org is a nearly six year old django application that has not been updated in more than three years, and it has simply atrophied in the face of the continuing march of progress. In short, it badly needs an update. Unfortunately, sr.org is one of those things that people love to use, but attracts no developer attention to love. Maybe it is time to decommission it with a note stating that it needs developer resources to bring it forward.

Howard
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Frank_Warmerdam · June 11, 2013, 4:09pm

Folks,

I will note there has been interest in helping from several fronts - most recently Aaron Racicot suggested some folks from CUGOS would like to help. I was also in contact with some europeans but that got hung up on some idea of needing a new VM and now I can’t seem to find the email thread with them. I think part of the challenge is enabling those who want to help. Spatialreference.org does need some love.

Best regards,

···

On Tue, Jun 11, 2013 at 9:00 AM, Howard Butler <hobu.inc@gmail.com> wrote:

… notes on spatialreference.org breaking due to system upgrades on the SAC Projects VM …

I regret that I might have had a tone that implied whomever made changes to the project machine’s django/python was to blame for things breaking. The reality is that spatialreference.org is a nearly six year old django application that has not been updated in more than three years, and it has simply atrophied in the face of the continuing march of progress. In short, it badly needs an update. Unfortunately, sr.org is one of those things that people love to use, but attracts no developer attention to love. Maybe it is time to decommission it with a note stating that it needs developer resources to bring it forward.

Howard

MetaCRS mailing list
MetaCRS@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/metacrs

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

Frank_Warmerdam · June 11, 2013, 4:25pm

Bobb,

It is now a subproject of the OSGeo Metacrs project and it hosted on OSGeo hardware. I think Howard Butler, Chris Schmidt, Dane Springmeyer and Josh Livni were the original authors (forgive me if I’m wrong).

Best regards,
Frank

···

On Tue, Jun 11, 2013 at 9:04 AM, Basques, Bob (CI-StPaul) <bob.basques@ci.stpaul.mn.us> wrote:

Howard,

Who owns spatialref.org?

I use it quite a bit myself, but it usually is the sort of thing where I need it now!! for a very tight timeline sort of thing. I often wondered about integrating it into something as a helper service of some sort, as well as extending the service a bit.

Bobb

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Howard Butler
Sent: Tuesday, June 11, 2013 11:01 AM
To: System Administration Committee Discussion/OSGeo
Cc: metacrs@lists.osgeo.org
Subject: Re: [SAC] spatialref.org

… notes on spatialreference.org breaking due to system upgrades on the SAC Projects VM …

I regret that I might have had a tone that implied whomever made changes to the project machine’s django/python was to blame for things breaking. The reality is that spatialreference.org is a nearly six year old django application that has not been updated in more than three years, and it has simply atrophied in the face of the continuing march of progress. In short, it badly needs an update. Unfortunately, sr.org is one of those things that people love to use, but attracts no developer attention to love. Maybe it is time to decommission it with a note stating that it needs developer resources to bring it forward.

Howard

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

hobu · June 11, 2013, 4:28pm

On Jun 11, 2013, at 11:25 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Bobb,

It is now a subproject of the OSGeo Metacrs project and it hosted on OSGeo hardware. I think Howard Butler, Chris Schmidt, Dane Springmeyer and Josh Livni were the original authors (forgive me if I'm wrong).

Yup, I transferred the domain name to OSGeo a few years ago. It's there for anyone willing to step in it, but as I said, it needs more than a new coat of paint.

Howard

Basques_Bob_CI-StPau · June 11, 2013, 4:44pm

All,

I'm willing to help on this, but need to share some recent experiences as a disclaimer (of sorts) . . .

Well first, since it's a service (most importantly) I would think putting it on it's own VM/hardware (well actually Hardware/VM is my preferred order, read on) would be the first thing to do. Services should not go down or get broken because of infrastructure changes to a service (in my mind). I tend to opt for leaving a service alone once it's up and running, it can be augmented, but any sort of major redo suggests setting up the service all over again, which sounds like where the SR.org site is at in it's lifecycle.

I'm doing something similar here with a 7 year old system as a matter of fact, it's run just fine over the years, in a 24/7 mode, going down a total of three times while on it's own hardware, all three times being network related and beyond my control. I'm much more comfortable with using a new piece of hardware instead of being at the mercy of a VM farm, which seems to go up/down almost weekly for maintenance of this or that, and which I have little control over.

In the interests of stability, I tend to use dedicated hardware for (each) service, does it cost more (I would argue no, based on the longevity of a service).

Ok, that's off my chest, so do you still want me to help :c)

Can someone shoot me the last couple of years of usage logs, or even a usage summary of the site? I may have some options for homing on dedicated space if that's of interest.

Bobb

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Howard Butler
Sent: Tuesday, June 11, 2013 11:29 AM
To: System Administration Committee Discussion/OSGeo
Subject: Re: [SAC] spatialref.org

On Jun 11, 2013, at 11:25 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Bobb,

It is now a subproject of the OSGeo Metacrs project and it hosted on OSGeo hardware. I think Howard Butler, Chris Schmidt, Dane Springmeyer and Josh Livni were the original authors (forgive me if I'm wrong).

Yup, I transferred the domain name to OSGeo a few years ago. It's there for anyone willing to step in it, but as I said, it needs more than a new coat of paint.

Howard

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Alex_M · June 11, 2013, 5:17pm

There is overhead to a 100% separate VM (Hardware and People). Leaving a system for 7+ years without security patches is also a recipe for it becoming subject to a botnet (recent trend is to hijack servers instead of desktops). I won't rule it out and suggest that if that's what we want someone needs to bug OSUOSL about the backup server we wanted (I've gotten no response lately) so that we free up space on osgeo4 for more VMs, or we approach the new EDU labs about load balance hosting (my lab would probably offer). We might also consider using CloudFlare at the free level to get world wide DOS protection and CDN(side benefit).

In terms of isolation, python virtualenv is a decent compromise, and tends to only break when python itself if upgraded. I had already isolated the wsgi component to gunicorn a few years ago, so that apache doesn't really effect it.

Such is the life of the projects VM. If we do feel we need more stability there are other VMs that see less changes, host fewer sites we could shift to.

FYI, from what I can tell, spatialreference.org is now Django 1.4.5 compatible. I spent about 1 hour making that happen which isn't terrible.

Thanks,
Alex

On 06/11/2013 09:44 AM, Basques, Bob (CI-StPaul) wrote:

All,

I'm willing to help on this, but need to share some recent experiences as a disclaimer (of sorts) . . .

Well first, since it's a service (most importantly) I would think putting it on it's own VM/hardware (well actually Hardware/VM is my preferred order, read on) would be the first thing to do. Services should not go down or get broken because of infrastructure changes to a service (in my mind). I tend to opt for leaving a service alone once it's up and running, it can be augmented, but any sort of major redo suggests setting up the service all over again, which sounds like where the SR.org site is at in it's lifecycle.

I'm doing something similar here with a 7 year old system as a matter of fact, it's run just fine over the years, in a 24/7 mode, going down a total of three times while on it's own hardware, all three times being network related and beyond my control. I'm much more comfortable with using a new piece of hardware instead of being at the mercy of a VM farm, which seems to go up/down almost weekly for maintenance of this or that, and which I have little control over.

In the interests of stability, I tend to use dedicated hardware for (each) service, does it cost more (I would argue no, based on the longevity of a service).

Ok, that's off my chest, so do you still want me to help :c)

Can someone shoot me the last couple of years of usage logs, or even a usage summary of the site? I may have some options for homing on dedicated space if that's of interest.

Bobb

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Howard Butler
Sent: Tuesday, June 11, 2013 11:29 AM
To: System Administration Committee Discussion/OSGeo
Subject: Re: [SAC] spatialref.org

On Jun 11, 2013, at 11:25 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Bobb,

It is now a subproject of the OSGeo Metacrs project and it hosted on OSGeo hardware. I think Howard Butler, Chris Schmidt, Dane Springmeyer and Josh Livni were the original authors (forgive me if I'm wrong).

Yup, I transferred the domain name to OSGeo a few years ago. It's there for anyone willing to step in it, but as I said, it needs more than a new coat of paint.

Howard

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Frank_Warmerdam · June 11, 2013, 5:46pm

Folks,

I would agree with Alex. I don’t see a pressing need to move sr.org off the projects vm at this time. I think the lowest hanging fruit is to just continue work on it in-situ.

For a bigger and more complex service having it’s own VM might make sense, but I don’t think that really applies here.

Best regards,

Frank

···

On Tue, Jun 11, 2013 at 10:17 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:

There is overhead to a 100% separate VM (Hardware and People). Leaving a system for 7+ years without security patches is also a recipe for it becoming subject to a botnet (recent trend is to hijack servers instead of desktops). I won’t rule it out and suggest that if that’s what we want someone needs to bug OSUOSL about the backup server we wanted (I’ve gotten no response lately) so that we free up space on osgeo4 for more VMs, or we approach the new EDU labs about load balance hosting (my lab would probably offer). We might also consider using CloudFlare at the free level to get world wide DOS protection and CDN(side benefit).

In terms of isolation, python virtualenv is a decent compromise, and tends to only break when python itself if upgraded. I had already isolated the wsgi component to gunicorn a few years ago, so that apache doesn’t really effect it.

Such is the life of the projects VM. If we do feel we need more stability there are other VMs that see less changes, host fewer sites we could shift to.

FYI, from what I can tell, spatialreference.org is now Django 1.4.5 compatible. I spent about 1 hour making that happen which isn’t terrible.

Thanks,
Alex

On 06/11/2013 09:44 AM, Basques, Bob (CI-StPaul) wrote:

All,

I’m willing to help on this, but need to share some recent experiences as a disclaimer (of sorts) . . .

Well first, since it’s a service (most importantly) I would think putting it on it’s own VM/hardware (well actually Hardware/VM is my preferred order, read on) would be the first thing to do. Services should not go down or get broken because of infrastructure changes to a service (in my mind). I tend to opt for leaving a service alone once it’s up and running, it can be augmented, but any sort of major redo suggests setting up the service all over again, which sounds like where the SR.org site is at in it’s lifecycle.

I’m doing something similar here with a 7 year old system as a matter of fact, it’s run just fine over the years, in a 24/7 mode, going down a total of three times while on it’s own hardware, all three times being network related and beyond my control. I’m much more comfortable with using a new piece of hardware instead of being at the mercy of a VM farm, which seems to go up/down almost weekly for maintenance of this or that, and which I have little control over.

In the interests of stability, I tend to use dedicated hardware for (each) service, does it cost more (I would argue no, based on the longevity of a service).

Ok, that’s off my chest, so do you still want me to help :c)

Can someone shoot me the last couple of years of usage logs, or even a usage summary of the site? I may have some options for homing on dedicated space if that’s of interest.

Bobb

-----Original Message-----
From: sac-bounces@lists.osgeo.org [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Howard Butler
Sent: Tuesday, June 11, 2013 11:29 AM
To: System Administration Committee Discussion/OSGeo
Subject: Re: [SAC] spatialref.org

On Jun 11, 2013, at 11:25 AM, Frank Warmerdam <warmerdam@pobox.com> wrote:

Bobb,

It is now a subproject of the OSGeo Metacrs project and it hosted on OSGeo hardware. I think Howard Butler, Chris Schmidt, Dane Springmeyer and Josh Livni were the original authors (forgive me if I’m wrong).

Yup, I transferred the domain name to OSGeo a few years ago. It’s there for anyone willing to step in it, but as I said, it needs more than a new coat of paint.

Howard

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

–
---------------------------------------±-------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

Martin_Spott · June 12, 2013, 11:21am

On Tue, Jun 11, 2013 at 10:46:52AM -0700, Frank Warmerdam wrote:

I would agree with Alex.

I do as well. And if we really get to the point to declare the sr.org
as unmaintainable (I hope we won't), I'd prefer to create an isolated
'package' containing its own Python, Django and the minimum set of
required Python modules to run on the "projects" VM rather than setting
up a distinct VM using an outdated environment.

Cheers,
Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------