#2934: osgeo7 systems went down
---------------------------+--------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Keywords:
---------------------------+--------------------------------------
osgeo7 went down and appeared to be down for about 15-20 minutes, though
osgeo7 and all the instances seemed fine.
All came back fine.
I'm suspecting it was a momentary issue with osgeo7 nginx, as trying to
run
{{{
systemctl status nginx
}}}
took longer than usual
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934>
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.
#2934: osgeo7 systems went down
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
I thought it was just osgeo7 but got failures on osgeo3 and osgeo9 as
well.
Given that I can ssh into the hosts and they don't seem to be taxed, I'm
guessing it's either an issue on my end, or something up with the PAIRS
dns.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:1>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo7 systems went down
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
Put in a support ticket with PAIRS. They confirmed its a DNS attack
against the osgeo.org domain, so that is why our other domains housed on
PAIRs were not affected unless they had a CNAME to osgeo.org. details
here
https://lists.osgeo.org/pipermail/sac/2023-May/015247.html
At the moment things seem to be back to normal, but I give it a day or so
before confirm it is all set.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:2>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo7 systems went down
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
No real change yet. They are still patching. I'll call again to ask if
they can move osgeo.org to separate NS servers. As DLange observed the
other domains we have are not on NS1 and NS2 pairdomains, but oare on
like NS3-NS8.
Then again foss4g is on NS1 and NS2 and haven't had trouble hitting them,
so if it is a DOS on particular set of domains that might not help.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:3>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):
* summary: osgeo7 systems went down => osgeo.org domains not reachable
Comment:
I called PAIRs again asking them to move our osgeo.org domain from NS1,
NS2 to NS3 - NS8 since the other domains we have on those domain servers
are not having issues.
They have that ticketed now but person on staff unfortunately is not
allowed to make the change since he's on the hosting instead of the DNS
side.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:4>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
Received an update from PAIRs note below
{{{
I wanted to follow up again. While the initial attack was mitigated, we
are still seeing many requests per second on what appears to be random
sub-domains of osgeo.org. The software to mitigate attacks has been
installed on all of our name server, and our system admins are monitoring
them.
Switching to different ns*.pairnic.com name servers would not help. Since
osgeo.org is being targeted, the attack will shift to whatever name
servers osgeo.org uses.
}}}
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:5>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
At several peoples requests, I've changed our DNS over to cloudflare.
Will monitor how this goes and close this out. I need to change the email
address on cloudflare, but I changed it too many times so am blocked at
the moment from changing it.
Once I change, I'll put the passwords and email account in passwordstore.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:6>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by jive):
A lot of things have improved! repo.osgeo.org is back and builds are green
One server that is not back yet is docs.geoserver.org, and geo-
docs.geoserver.org (used for uploading content).
I am not sure if this is related to the DNS trouble; of if this specific
instance is not yet happy.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:7>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: new
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution:
Keywords: |
---------------------------+---------------------------------------
Comment (by robe):
geo-docs.geoserver.org and docs.geoserver.org should be fixed now.
I think it's because I still had web.osgeo3.osgeo.org still in proxy mode.
I've removed that so it's DNS only now.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:8>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):
* status: new => closed
* resolution: => fixed
Comment:
Things still seem to be working. I'v committed the account info to the
password store.
@strk and @jef, please confirm you can read them.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:9>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
#2934: osgeo.org domains not reachable
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------------------------
Comment (by strk):
I confirm I'm able to read the password, thanks
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2934#comment:10>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.