[SAC] xblade14-2: Service Temporarily Unavailable (Apache) - gdal, grass, qgis, rso, mapbuilder

neteler · September 15, 2008, 8:31am

Hi SAC,

I have the problem that this page (which contains a series of screenshot
thumbnails) doesn't load completely - happens frequently to me on
that page:
http://grass.osgeo.org/screenshots/gui.php

Reloading it a few times lead to 503

Service Temporarily Unavailable

The server is temporarily unable to service your request due to
maintenance downtime or capacity problems. Please try again later.
Apache/2.0.54 (Fedora) Server at grass.osgeo.org Port 80

[root@xblade14-2 httpd]# tail grass_error_log
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:41 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/index.php
[Mon Sep 15 01:25:12 2008] [error] [client 141.223.191.205] Rejecting
client at 141.223.191.205, referer:
http://grass.osgeo.org/wiki/Main_Page
[Mon Sep 15 01:25:12 2008] [error] [client 141.223.191.205] Rejecting
client at 141.223.191.205, referer:
http://grass.osgeo.org/wiki/Main_Page
[root@xblade14-2 httpd]# host 87.5.230.95
95.230.5.87.in-addr.arpa domain name pointer
host95-230-dynamic.5-87-r.retail.telecomitalia.it.

It seems that Apache is severly suffering.
But the CPU(s) aren't doing much:

[neteler@xblade14-2 ~]$ w
01:24:49 up 47 days, 2:10, 1 user, load average: 0.33, 0.45, 0.61
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
neteler pts/0 host95-230-dynam 01:21 1.00s 0.05s 0.02s w

[neteler@xblade14-2 ~]$ top
top - 01:24:51 up 47 days, 2:10, 1 user, load average: 0.30, 0.45, 0.61
Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.2% us, 2.4% sy, 0.3% ni, 68.9% id, 7.0% wa, 0.2% hi, 0.0% si
Mem: 1034320k total, 1006748k used, 27572k free, 29760k buffers
Swap: 2096472k total, 136612k used, 1959860k free, 396052k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  850 neteler 15 0 4752 1520 1128 R 1.9 0.1 0:00.02 top
    1 root 16 0 1744 512 488 S 0.0 0.0 0:03.69 init
    2 root 34 19 0 0 0 S 0.0 0.0 0:02.16 ksoftirqd/0
...

[root@xblade14-2 httpd]# grep 'Rejecting client' grass_error_log | wc -l
4869

Any clue? Apache seems to be messed up:

[root@xblade14-2 httpd]# grep -l 'Rejecting client' *_error_log
gdal_error_log
grass_error_log
mapbuilder_error_log
qgis_error_log
rso_error_log

It affects all of us who are hosted there.

Markus

Daniel_Morissette · September 15, 2008, 3:12pm

Markus,

Could this be related to the connection throttling setup in http://trac.osgeo.org/osgeo/ticket/216 kicking in because the page generates too many connections?

Daniel

Markus Neteler wrote:

Hi SAC,

I have the problem that this page (which contains a series of screenshot
thumbnails) doesn't load completely - happens frequently to me on
that page:
http://grass.osgeo.org/screenshots/gui.php

Reloading it a few times lead to 503

Service Temporarily Unavailable

The server is temporarily unable to service your request due to
maintenance downtime or capacity problems. Please try again later.
Apache/2.0.54 (Fedora) Server at grass.osgeo.org Port 80

[root@xblade14-2 httpd]# tail grass_error_log
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:19 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/gui.php
[Mon Sep 15 01:24:41 2008] [error] [client 87.5.230.95] Rejecting
client at 87.5.230.95, referer:
http://grass.osgeo.org/screenshots/index.php
[Mon Sep 15 01:25:12 2008] [error] [client 141.223.191.205] Rejecting
client at 141.223.191.205, referer:
http://grass.osgeo.org/wiki/Main_Page
[Mon Sep 15 01:25:12 2008] [error] [client 141.223.191.205] Rejecting
client at 141.223.191.205, referer:
http://grass.osgeo.org/wiki/Main_Page
[root@xblade14-2 httpd]# host 87.5.230.95
95.230.5.87.in-addr.arpa domain name pointer
host95-230-dynamic.5-87-r.retail.telecomitalia.it.

It seems that Apache is severly suffering.
But the CPU(s) aren't doing much:

[neteler@xblade14-2 ~]$ w
01:24:49 up 47 days, 2:10, 1 user, load average: 0.33, 0.45, 0.61
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
neteler pts/0 host95-230-dynam 01:21 1.00s 0.05s 0.02s w

[neteler@xblade14-2 ~]$ top
top - 01:24:51 up 47 days, 2:10, 1 user, load average: 0.30, 0.45, 0.61
Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.2% us, 2.4% sy, 0.3% ni, 68.9% id, 7.0% wa, 0.2% hi, 0.0% si
Mem: 1034320k total, 1006748k used, 27572k free, 29760k buffers
Swap: 2096472k total, 136612k used, 1959860k free, 396052k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  850 neteler 15 0 4752 1520 1128 R 1.9 0.1 0:00.02 top
    1 root 16 0 1744 512 488 S 0.0 0.0 0:03.69 init
    2 root 34 19 0 0 0 S 0.0 0.0 0:02.16 ksoftirqd/0
...

[root@xblade14-2 httpd]# grep 'Rejecting client' grass_error_log | wc -l
4869

Any clue? Apache seems to be messed up:

[root@xblade14-2 httpd]# grep -l 'Rejecting client' *_error_log
gdal_error_log
grass_error_log
mapbuilder_error_log
qgis_error_log
rso_error_log

It affects all of us who are hosted there.

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

--
Daniel Morissette
http://www.mapgears.com/

neteler · September 15, 2008, 4:33pm

On Mon, Sep 15, 2008 at 5:12 PM, Daniel Morissette
<dmorissette@mapgears.com> wrote:

Markus,

Could this be related to the connection throttling setup in
http://trac.osgeo.org/osgeo/ticket/216 kicking in because the page generates
too many connections?

ouch. Since I guess that I suggested this mod_cband tool...

Can we please permit more bandwidth? I consider my manual click
rate as pretty low frequency...

thanks
Markus

Jason_Birch · September 15, 2008, 8:36pm

It's not really the page's fault, your network stack is configured contrary to the recommendations of RFC2068 section 8.1.4, or there is a proxy between you and the osgeo server with more than one user connected to the same site:

Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD maintain AT MOST 2 connections with any
   server or proxy. A proxy SHOULD use up to 2*N connections to another
   server or proxy, where N is the number of simultaneously active
   users.

Perhaps 3 connections is a bit low for a general purpose website though. Is it possible to configure the grass site differently than the downloads?

Jason

neteler · September 15, 2008, 9:08pm

On Mon, Sep 15, 2008 at 10:36 PM, Jason Birch <Jason.Birch@nanaimo.ca> wrote:

It's not really the page's fault,

Our web page(s) used to work for years...

your network stack is configured contrary to the recommendations of RFC2068 section 8.1.4,
or there is a proxy between you and the osgeo server with more than one user connected to the same site:

I am directly connected via 20MBit/s (ok, maybe 15Mbit/s effectively) ADSL.
I observe that the server got significant slower recently, I used to reach
almost local network saturation while I get a fraction of this these days.

Clients that use persistent connections SHOULD limit the number of
  simultaneous connections that they maintain to a given server. A
  single-user client SHOULD maintain AT MOST 2 connections with any
  server or proxy. A proxy SHOULD use up to 2*N connections to another
  server or proxy, where N is the number of simultaneously active
  users.

Perhaps 3 connections is a bit low for a general purpose website though.
Is it possible to configure the grass site differently than the downloads?

This would be great. Bandwidth is there AFAIK.

thanks,
Markus

Christopher_Schmidt1 · September 15, 2008, 10:08pm

On Mon, Sep 15, 2008 at 11:08:03PM +0200, Markus Neteler wrote:

On Mon, Sep 15, 2008 at 10:36 PM, Jason Birch <Jason.Birch@nanaimo.ca> wrote:
> It's not really the page's fault,

Our web page(s) used to work for years...

> your network stack is configured contrary to the recommendations of RFC2068 section 8.1.4,
> or there is a proxy between you and the osgeo server with more than one user connected to the same site:

I am directly connected via 20MBit/s (ok, maybe 15Mbit/s effectively) ADSL.
I observe that the server got significant slower recently, I used to reach
almost local network saturation while I get a fraction of this these days.

> Clients that use persistent connections SHOULD limit the number of
> simultaneous connections that they maintain to a given server. A
> single-user client SHOULD maintain AT MOST 2 connections with any
> server or proxy. A proxy SHOULD use up to 2*N connections to another
> server or proxy, where N is the number of simultaneously active
> users.
>
> Perhaps 3 connections is a bit low for a general purpose website though.
> Is it possible to configure the grass site differently than the downloads?

This would be great. Bandwidth is there AFAIK.

It's not (and never was) about Bandwidth. It's about tying up apache
children, which each use memory, typically with download accelerators,
which then makes the entire site sluggish since other users can't get
access to the site because there's no remaining apache children.

If mod_cband can be configured this way, I don't have a problem with it,
but I wanted to clarify that bandwidth has never been the problem.

Regards,
--
Christopher Schmidt
MetaCarta