[SAC] trac.osgeo.org with load average of 24

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

Markus

On Sat, Nov 05, 2016 at 06:15:01PM +0100, Markus Neteler wrote:

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

I'm no expert, but whenever I tried to debug slowness I found it
useless that top(1) only shows "apache" w/out a way to tell which
service is actually keeping apache busy.

I suspect it's trac, being a big apache module, but I've no proof
of that.

One thing I know is that I enabled trac logging when I started looking
at trac, so it could be tried to disable it, see /etc/trac/* (it's
under git).

--strk;

On Sat, Nov 5, 2016 at 6:49 PM, Sandro Santilli <strk@kbt.io> wrote:

On Sat, Nov 05, 2016 at 06:15:01PM +0100, Markus Neteler wrote:

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

I'm no expert, but whenever I tried to debug slowness I found it
useless that top(1) only shows "apache" w/out a way to tell which
service is actually keeping apache busy.

I suspect it's trac, being a big apache module, but I've no proof
of that.

Looks like that, as it can be seen with strace:

strace -p `ps aux | grep apache | head -n 2 | tail -n 1 | cut -d' ' -f2`

Markus

Are we still running with mod_python or mod_wsgi? Might be time to move
to uwsgi or gunicorn with supervisord. They scale better.

Apache has an internal status page. This makes me wonder what email the
munin monitoring is supposed to hit and if we can setup a rule for load
to trigger email to SAC.

-Alex

On 11/05/2016 10:57 AM, Markus Neteler wrote:

On Sat, Nov 5, 2016 at 6:49 PM, Sandro Santilli <strk@kbt.io> wrote:

On Sat, Nov 05, 2016 at 06:15:01PM +0100, Markus Neteler wrote:

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

I'm no expert, but whenever I tried to debug slowness I found it
useless that top(1) only shows "apache" w/out a way to tell which
service is actually keeping apache busy.

I suspect it's trac, being a big apache module, but I've no proof
of that.

Looks like that, as it can be seen with strace:

strace -p `ps aux | grep apache | head -n 2 | tail -n 1 | cut -d' ' -f2`

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

On Sat, Nov 05, 2016 at 06:57:39PM +0100, Markus Neteler wrote:

strace -p `ps aux | grep apache | head -n 2 | tail -n 1 | cut -d' ' -f2`

It took 'awk' here instead of 'cut' as the last snippet, but
thanks !

I see many lines like these:

  stat("/usr/local/lib/python2.6/dist-packages/TracMetrixPlugin-0.1.8dev-py2.6.egg/subprocess", 0x7ffe903d3950) = -1 ENOENT (No such file or directory)
  open("/usr/local/lib/python2.6/dist-packages/TracMetrixPlugin-0.1.8dev-py2.6.egg/subprocess.so", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/usr/local/lib/python2.6/dist-packages/TracMetrixPlugin-0.1.8dev-py2.6.egg/subprocessmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/usr/local/lib/python2.6/dist-packages/TracMetrixPlugin-0.1.8dev-py2.6.egg/subprocess.py", O_RDONLY) = -1 ENOENT (No such file or directory)
  open("/usr/local/lib/python2.6/dist-packages/TracMetrixPlugin-0.1.8dev-py2.6.egg/subprocess.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
  stat("/usr/local/lib/python2.6/dist-packages/TracSectionEditPlugin-0.2.6-py2.6.egg/subprocess", 0x7ffe903d3950) = -1 ENOENT (No such file or directory)
  open("/usr/local/lib/python2.6/dist-packages/TracSectionEditPlugin-0.2.6-py2.6.egg/subprocess.so", O_RDONLY) = -1 ENOENT (No such file or directory)

As if trac needs to find files over and over again, shouldn't it be loaded
all in memory at once instead ? Maybe it's still running as a pure cgi ?

--strk;

On Sat, Nov 05, 2016 at 11:32:46AM -0700, Alex Mandel wrote:

Are we still running with mod_python or mod_wsgi? Might be time to move
to uwsgi or gunicorn with supervisord. They scale better.

Apache is using mod_python,
according to /etc/apache2/includes/trac_env.inc
which is included by every instance.

Apache has an internal status page.

How to access it ?

This makes me wonder what email the
munin monitoring is supposed to hit and if we can setup a rule for load
to trigger email to SAC.

No munin experience here, sorry

--strk;

Le samedi 05 novembre 2016 18:15:01, Markus Neteler a écrit :

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

Hi,

I'm not sure if it is related but the GDAL Travis-CI instances experience a
lot of flake failures during the execution of the automated tests due to
timeouts on download.osgeo.org

ERROR 11: CURL error: Failed to connect to download.osgeo.org port 80:
Connection timed out
gdalinfo failed - unable to open
'/vsicurl/http://download.osgeo.org/gdal/data/ecw/spif83.ecw’.

I'm also experiencing locally very slow downloading on this file.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

On Sat, Nov 05, 2016 at 08:19:39PM +0100, Sandro Santilli wrote:

On Sat, Nov 05, 2016 at 11:32:46AM -0700, Alex Mandel wrote:
> Are we still running with mod_python or mod_wsgi? Might be time to move
> to uwsgi or gunicorn with supervisord. They scale better.

Apache is using mod_python,
according to /etc/apache2/includes/trac_env.inc
which is included by every instance.

> Apache has an internal status page.

How to access it ?

Getting back to this.

Alex, did you receive my mail ?
Anyone knows what's the internal status page he's talking about ?

--strk;

On Nov 10, 2016 7:36 PM, “Sandro Santilli” <strk@kbt.io> wrote:

On Sat, Nov 05, 2016 at 08:19:39PM +0100, Sandro Santilli wrote:

On Sat, Nov 05, 2016 at 11:32:46AM -0700, Alex Mandel wrote:

Are we still running with mod_python or mod_wsgi? Might be time to move
to uwsgi or gunicorn with supervisord. They scale better.

Apache is using mod_python,
according to /etc/apache2/includes/trac_env.inc
which is included by every instance.

Apache has an internal status page.

How to access it ?

Getting back to this.

Alex, did you receive my mail ?
Anyone knows what’s the internal status page he’s talking about ?

Perhaps this one?

http://trac.osgeo.org/server-status

Markus

–strk;


Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

On Thu, Nov 10, 2016 at 07:45:04PM +0100, Markus Neteler wrote:

On Nov 10, 2016 7:36 PM, "Sandro Santilli" <strk@kbt.io> wrote:
> On Sat, Nov 05, 2016 at 08:19:39PM +0100, Sandro Santilli wrote:
> > On Sat, Nov 05, 2016 at 11:32:46AM -0700, Alex Mandel wrote:
> > >
> > > Apache has an internal status page.
> >
> > How to access it ?
>
> Getting back to this.
>
> Alex, did you receive my mail ?
> Anyone knows what's the internal status page he's talking about ?

Perhaps this one?

http://trac.osgeo.org/server-status

Forbidden.

PS: does anyone know how to contact Alex in real-life ?

--strk;

On 11/16/2016 08:25 AM, Sandro Santilli wrote:

On Thu, Nov 10, 2016 at 07:45:04PM +0100, Markus Neteler wrote:

On Nov 10, 2016 7:36 PM, "Sandro Santilli" <strk@kbt.io> wrote:

On Sat, Nov 05, 2016 at 08:19:39PM +0100, Sandro Santilli wrote:

On Sat, Nov 05, 2016 at 11:32:46AM -0700, Alex Mandel wrote:

Apache has an internal status page.

How to access it ?

Getting back to this.

Alex, did you receive my mail ?
Anyone knows what's the internal status page he's talking about ?

Perhaps this one?

http://trac.osgeo.org/server-status

Forbidden.

PS: does anyone know how to contact Alex in real-life ?

--strk;

I am trying to keep up with the volume of email. The internal status
page for security reasons is likely only accessible via localhost. This
is standard Apache stuff. I stand by my recommendation to simply move
from mod_python to other methods of running trac. Though checking to
make sure the load is legitimate is good.

Thanks,
Alex

Trac seems down for me again.

-jeff

On 2016-11-05 2:15 PM, Markus Neteler wrote:

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

Markus

Resolved.

-jeff

On 2016-12-17 12:44 PM, Jeff McKenna wrote:

Trac seems down for me again.

-jeff

On 2016-11-05 2:15 PM, Markus Neteler wrote:

Hi,

for several days (longer?), the load average on trac is very high -
leading to slow response rates when using the server.

Any ideas how to optimize apache?

Markus