#3102: Getting random 500s on osgeo7 hosts
----------------------+--------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Keywords:
----------------------+--------------------------------------
Just noticed in past 5 minutes
OSGeo
OSGeo Git Services: Gitea - Git with a cup of tea
https://photoprism.osgeo.org
https://download.osgeo.org
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/3102>
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#3102: Getting random 500s on osgeo7 hosts
----------------------+---------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Resolution: fixed
Keywords: |
----------------------+---------------------------------------
Changes (by robe):
* status: new => closed
* resolution: => fixed
Comment:
Seems to be fine now. I rebooted nginx after doing some patch updates.
The prometheus nginx exporter was failing because of apparmor.
I haven't figured out what settings to do with apparmor and what benefit
it has for us exactly so I disabled it but seems to have started back on
it's own, so doesn't stay disabled
{{{
systemctl disable apparmor
}}}
I suspect if I reboot the nginx-prometheus will stop working again. Not a
huge deal.
--
Ticket URL: <#3102 (Getting random 500s on osgeo7 hosts) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#3102: Getting random 500s on osgeo7 hosts
----------------------+---------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: reopened
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Resolution:
Keywords: |
----------------------+---------------------------------------
Changes (by robe):
* status: closed => reopened
* resolution: fixed =>
Comment:
Still seems to be an issue but just on osgeo7 sites.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/3102#comment:2>
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#3102: Getting random 500s on osgeo7 hosts
----------------------+---------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: reopened
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Resolution:
Keywords: |
----------------------+---------------------------------------
Comment (by robe):
Okay I think there is some extra traffic happening cause the
/var/log/nginx/error.log was showing errors
{{{
768 worker_connections not enough
}}}
So it was going up and down.
I've upped that 1024, but 768 was kinda low, so surprised we haven't run
into issues with this before. I'm going to check our other nginx.
I also just deleted that rewrite file in /etc/nginx/sites-enabled as I
suspect those have all been moved to osgeo9 and they are mostly redirects
for .osgeo.org sites that redirect to www.osgeo.org
I'm going to keep this open for now until I have reviewed the situation
more.
--
Ticket URL: <#3102 (Getting random 500s on osgeo7 hosts) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#3102: Getting random 500s on osgeo7 hosts
----------------------+---------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: reopened
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Resolution:
Keywords: |
----------------------+---------------------------------------
Comment (by robe):
oops this could be related to my removal of spatialreference.org in #3101
I'm see a lot of errors in the error logs of sites trying to hit
www.spatialreference.org which maybe a cache issue cause
www.spatialreference.org when I ping does go to
cdn-185-199-108-153.github.com
--
Ticket URL: <#3102 (Getting random 500s on osgeo7 hosts) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#3102: Getting random 500s on osgeo7 hosts
----------------------+---------------------------------------
Reporter: robe | Owner: sac-tickets@…
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2024-I
Component: SysAdmin | Resolution: fixed
Keywords: |
----------------------+---------------------------------------
Changes (by robe):
* status: reopened => closed
* resolution: => fixed
Comment:
This seems to have dissipated after increasing max worker_processes and
haven't seen these 500s in the past 6 hrs.
--
Ticket URL: <#3102 (Getting random 500s on osgeo7 hosts) – OSGeo;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.