[OSGeo] #3462: Looks like grasswiki is down

Committees SAC tickets
1

#3462: Looks like grasswiki is down
----------------------+---------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: new
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Keywords:
----------------------+---------------------------
giving this error:

{{{
#0 /var/www/grass/grass-
wiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(805):
Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /var/www/grass/grass-
wiki/w/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793):
Wikimedia\Rdbms\LoadBalancer->getServerConnection()
#2 /var/www/grass/grass-
wiki/w/includes/libs/rdbms/database/DBConnRef.php(103):
Wikimedia\Rdbms\LoadBalancer->getConnectionInternal()
#3 /var/www/grass/grass-
wiki/w/includes/libs/rdbms/database/DBConnRef.php(117):
Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /var/www/grass/grass-
wiki/w/includes/libs/rdbms/database/DBConnRef.php(538):
Wikimedia\Rdbms\DBConnRef->__call()
#5 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(597):
Wikimedia\Rdbms\DBConnRef->anyString()
#6 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(550):
MessageCache->loadFromDB()
#7 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(446):
MessageCache->loadFromDBWithLocalLock()
#8 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(341):
MessageCache->loadUnguarded()
#9 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(1301):
MessageCache->load()
#10 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(1206):
MessageCache->getMsgFromNamespace()
#11 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(1177):
MessageCache->getMessageForLang()
#12 /var/www/grass/grass-wiki/w/includes/language/MessageCache.php(1075):
MessageCache->getMessageFromFallbackChain()
#13 /var/www/grass/grass-wiki/w/includes/language/Message.php(1485):
MessageCache->get()
#14 /var/www/grass/grass-wiki/w/includes/language/Message.php(972):
Message->fetchMessage()
#15 /var/www/grass/grass-wiki/w/includes/language/Message.php(1059):
Message->format()
#16 /var/www/grass/grass-wiki/w/includes/title/Title.php(695):
Message->text()
#17 /var/www/grass/grass-wiki/w/includes/MediaWiki.php(169):
MediaWiki\Title\Title::newMainPage()
#18 /var/www/grass/grass-wiki/w/includes/MediaWiki.php(189):
MediaWiki->parseTitle()
#19 /var/www/grass/grass-wiki/w/includes/MediaWiki.php(908):
MediaWiki->getTitle()
#20 /var/www/grass/grass-wiki/w/includes/MediaWiki.php(613):
MediaWiki->main()
#21 /var/www/grass/grass-wiki/w/index.php(50): MediaWiki->run()
#22 /var/www/grass/grass-wiki/w/index.php(46): wfIndexMain()
#23 {main}
}}}
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

2

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: closed
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution: fixed
Keywords: |
----------------------+----------------------------
Changes (by robe):

* resolution: => fixed
* status: new => closed

Comment:

it looks like mysql is running but hitting a too many connections error.

{{{
Nov 06 23:50:00 grass-wiki mariadbd[1837209]: 2025-11-06 23:50:00 0
[Warning] Aborted connection 0 to db: 'unconnected' user:
'unauthenticated' host: 'connecting host' (Too many connections)
Nov 06 23:50:00 grass-wiki mariadbd[1837209]: 2025-11-06 23:50:00 0
[Warning] Aborted connection 0 to db: 'unconnected' user:
'unauthenticated' host: 'connecting host' (Too many connections)
Nov 06 23:50:00 grass-wiki mariadbd[1837209]: 2025-11-06 23:50:00 0
[Warning] Aborted connection 0 to db: 'unconnected' user:
'unauthenticated' host: 'connecting host' (Too many connections)
}}}

So perhaps some sort of bot attack. I restarted the grasswiki container
and site came back but seems a bit sluggish. Will close this for now but
reopen if it happens again.
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

3

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Changes (by robe):

* resolution: fixed =>
* status: closed => reopened

Comment:

I've been getting a lot of notices of it going up and down so I suspect
some sort of bot attack.
But I haven't looked to confirm.
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

4

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

I checked and it is a botnet.

I blocked some IPs with iptables which helped (for some hour?) then ran
our of time.

I'll try to implement a fail2ban rule.
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

5

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

I have put a fail2ban mediawiki-ddos filter in place:

{{{
# after 3 min of filter activity

fail2ban-client status mediawiki-ddos
Status for the jail: mediawiki-ddos
|- Filter
| |- Currently failed: 4
| |- Total failed: 2974
| `- File list: /var/log/apache2/grasswiki.osgeo.org_access.log
`- Actions
    |- Currently banned: 34
    |- Total banned: 34
    `- Banned IP list: 47.82.15.1 47.82.14.196 47.82.15.217 47.79.51.190
47.82.15.49 47.82.15.242 47.79.51.127 47.82.14.138 47.82.13.222
47.82.14.67 47.82.13.125 47.82.15.20 47.79.51.110 47.82.13.216
47.79.51.119 47.82.13.83 47.79.51.170 47.82.14.40 47.82.14.136 47.79.51.57
47.82.13.67 47.82.13.95 47.79.51.95 47.82.14.209 47.82.15.144
17.241.75.211 47.82.15.74 47.82.13.208 47.82.14.174 47.82.13.159
47.79.51.152 17.22.253.187 17.22.237.115 17.241.219.51
}}}

Let's see if it improves the accessibility of
https://grasswiki.osgeo.org/wiki/
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

6

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

I have put very tight fail2ban settings which seem to help:

{{{
fail2ban-client status mediawiki-ddos | grep banned
grass-wiki: Fri Nov 14 10:28:11 2025

    |- Currently banned: 167
    |- Total banned: 378
}}}

Apache workers (before > 100):

{{{
ps aux | grep apache | wc -l
grass-wiki: Fri Nov 14 10:28:19 2025
15
}}}

Memory footprint:

{{{
free -h
                total used free shared buff/cache
available
Mem: 3.7Gi 381Mi 3.2Gi 39Mi 110Mi
3.3Gi
Swap: 1.9Gi 4.0Mi 1.9Gi

top
top - 10:32:25 up 7 days, 10:37, 3 users, load average: 20.75, 24.70,
30.00
Tasks: 66 total, 2 running, 63 sleeping, 0 stopped, 1 zombie
%Cpu(s): 30.5 us, 4.2 sy, 0.0 ni, 65.3 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
MiB Mem : 3814.7 total, 3178.5 free, 525.9 used, 110.3
buff/cache
MiB Swap: 1952.0 total, 1947.8 free, 4.2 used. 3249.2 avail Mem
}}}

@robe: could the RAM be increased by 2GB (from 4GB to 6 GB)? That would
avoid swapping.

The site is still slow, though, perhaps due to the high load average? Any
other DDOS-ing on the same host ongoing?
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

7

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

The Wiki is now much faster again.

Current stats after my filter update:

{{{
neteler@grass-wiki:~$ sudo fail2ban-client status mediawiki-ddos
[sudo] password for neteler:
Status for the jail: mediawiki-ddos
|- Filter
| |- Currently failed: 3
| |- Total failed: 76575
| `- File list: /var/log/apache2/grasswiki.osgeo.org_access.log
`- Actions
    |- Currently banned: 633
    |- Total banned: 1320
    `- Banned IP list: [...]
}}}

A lot IPs got banned :slight_smile:
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

8

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

Update, after hardening the `fail2ban` settings:

{{{
    |- Total banned: 9251
}}}
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

9

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by robe):

@neteler. No other ddos on the host to my knowledge but I haven't looked
that closely. Most other things require log in to view anything so
probably less prone to DDOS.

You still want RAM increased or you good after the changes you made?
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.

10

#3462: Looks like grasswiki is down
----------------------+----------------------------
Reporter: robe | Owner: sac-tickets@…
     Type: task | Status: reopened
Priority: normal | Milestone: Unplanned
Component: SysAdmin | Resolution:
Keywords: |
----------------------+----------------------------
Comment (by neteler):

Probably we are good for now. The load average is down to < 15 and the
Wiki is responsive.
--
Ticket URL: <Making sure you're not a bot!;
OSGeo <Gter - OSGeo;
OSGeo committee and general foundation issue tracker.