[SAC] Legit users vs. spam users

darkblueb · November 24, 2015, 1:10am

Hi Markus, Christian, Martin --
I have contacts at the Wikimedia Foundation, since last year.. here is the chat log just now:

  dbb hi all - our heavily used (and beloved) MediaWiki just got upgraded, and its time to take a look at the outrageous number of spam users being created daily, and spam pages
   dbb I believe I will get access at every level, but I want to be careful..
   dbb .. just looking into it now
   dbb !!! the sys-admin mailing list says 16300 spam user accounts created in the last four days ?!?!?
   wm-bot There is no such key, you probably want to try: !bribe, !tss,
   dbb "UserMerge and BlockAndNuke have failed" .. it says here
   dbb f***
   MaxSem !spam
   wm-bot For information about combating and handling spam in MediaWiki, see <http://www.mediawiki.org/wiki/Manual:Combating_spam> and <http://www.mediawiki.org/wiki/Anti-spam_features>\.
   dbb oh - MaxSem .. hi .. Brian at OSGeo here
   dbb I have a mess on my hands apparently
   wmat dbb: are they actually creating content?
   wmat dbb: if not, just run removeUnusedAccounts.php to delete all of them
   MaxSem 1) disable account creation until more measures are taken
   dbb I have just opened this minutes ago
   dbb yes agree - this is crazy
   wmat dbb: I use a dynamic captcha + confirm account + I blow away unused accounts once a month
   dbb we had a tech add LDAP login very recently, and upgrade the version
   dbb we use our LDAP heavily for other sites
   MaxSem I don't see LDAP auth at http://wiki.osgeo.org/wiki/Special:Version
   anchit
   MaxSem anchit, mentor? are you a developer?
   MaxSem dbb, meanwhile +2 accounts
   dbb ugh
   dbb ..
   MaxSem disable creation, then figure out what to do
   dbb whats the best way to disable ?
   MaxSem !access
   wm-bot For information on customizing user access, see <http://www.mediawiki.org/wiki/Manual:User_rights>\. For common examples of restricting access using both rights and extensions, see <http://www.mediawiki.org/wiki/Manual:Preventing_access>\.

in other words, $wgGroupPermissions['*']['createaccount'] = false;

--
Brian M Hamlin
OSGeo California Chapter
blog.light42.com

neteler · November 24, 2015, 9:08am

Hi Brian,

On Tue, Nov 24, 2015 at 2:10 AM, Brian M Hamlin <maplabs@light42.com> wrote:

Hi Markus, Christian, Martin --
I have contacts at the Wikimedia Foundation, since last year.. here is the chat log just now:

        dbb hi all - our heavily used (and beloved) MediaWiki just got upgraded, and its time to take a look at the outrageous number of spam users being created daily, and spam pages
                        dbb I believe I will get access at every level, but I want to be careful..
                        dbb .. just looking into it now
                        dbb !!! the sys-admin mailing list says 16300 spam user accounts created in the last four days ?!?!?

--> no, maybe over the past 2 years.

But since the recent updates it gots way worth.

I may note that in the previous version I made a local modification to
ConfirmEdit which notably decreased the amount of spam.
This was not (yet) migrated to the actual installation we have.

wm-bot There is no such key, you probably want to try: !bribe, !tss,
dbb "UserMerge and BlockAndNuke have failed" .. it says here

I have installed both.
But BlockAndNuke ignores the whitelist.txt which is rather annoying.

Question for them:
I have a list of 2800 blocked accounts but how to delete them?

                        dbb f***
                        MaxSem !spam
                        wm-bot For information about combating and handling spam in MediaWiki, see <http://www.mediawiki.org/wiki/Manual:Combating_spam> and <http://www.mediawiki.org/wiki/Anti-spam_features>\.

... I studies these pages for a few hours...

                        dbb oh - MaxSem .. hi .. Brian at OSGeo here
                        dbb I have a mess on my hands apparently
                        wmat dbb: are they actually creating content?
                        wmat dbb: if not, just run removeUnusedAccounts.php to delete all of them

Done but they don't disappear from the newusers log wiki page. Why??

MaxSem 1) disable account creation until more measures are taken

... probably a good idea.

                        dbb I have just opened this minutes ago
                        dbb yes agree - this is crazy
                        wmat dbb: I use a dynamic captcha + confirm account + I blow away unused accounts once a month

I suspect that they spamers/spam bots bypass all this and inject code somewhere.

dbb we had a tech add LDAP login very recently, and upgrade the version
dbb we use our LDAP heavily for other sites

Please teach us how to connect our Wiki to LDAP. Martin Spott could
not figure that out.

MaxSem I don't see LDAP auth at http://wiki.osgeo.org/wiki/Special:Version

Yes, we don't have it yet since we don't know how to implement the
connection between our Wiki and our LDAP.

                        anchit
                        MaxSem anchit, mentor? are you a developer?
                        MaxSem dbb, meanwhile +2 accounts
                        dbb ugh
                        dbb ..
                        MaxSem disable creation, then figure out what to do
                        dbb whats the best way to disable ?
                        MaxSem !access
                        wm-bot For information on customizing user access, see <http://www.mediawiki.org/wiki/Manual:User_rights>\. For common examples of restricting access using both rights and extensions, see <http://www.mediawiki.org/wiki/Manual:Preventing_access>\.

                        in other words, $wgGroupPermissions['*']['createaccount'] = false;

Time to switch that on?

Markus

Alex_M · November 24, 2015, 3:12pm

On 11/24/2015 01:08 AM, Markus Neteler wrote:

                        dbb we had a tech add LDAP login very recently, and upgrade the version
                        dbb we use our LDAP heavily for other sites

Please teach us how to connect our Wiki to LDAP. Martin Spott could
not figure that out.

                        MaxSem I don't see LDAP auth at http://wiki.osgeo.org/wiki/Special:Version

Yes, we don't have it yet since we don't know how to implement the
connection between our Wiki and our LDAP.

                        anchit
                        MaxSem anchit, mentor? are you a developer?
                        MaxSem dbb, meanwhile +2 accounts
                        dbb ugh
                        dbb ..
                        MaxSem disable creation, then figure out what to do
                        dbb whats the best way to disable ?
                        MaxSem !access
                        wm-bot For information on customizing user access, see <http://www.mediawiki.org/wiki/Manual:User_rights>\. For common examples of restricting access using both rights and extensions, see <http://www.mediawiki.org/wiki/Manual:Preventing_access>\.

                        in other words, $wgGroupPermissions['*']['createaccount'] = false;

Time to switch that on?

Markus

The only tricky part with LDAP is that we need some way for people to
link their LDAP login with existing accounts (and they need to know the
password for both).

https://www.mediawiki.org/wiki/Extension:LDAP_Authentication

Alex

darkblueb · November 24, 2015, 7:49pm

a quick look shows:

http://git.wikimedia.org/summary/?r=mediawiki/extensions/LDAPAuthorization.git

http://git.wikimedia.org/summary/?r=mediawiki/extensions/LdapAuthentication.git

asking on IRC, the second one may “do a lot more” …

Is this an indication that we should pay for experienced MediaWiki consulting here ?

–Brian

Christian_Willmes · November 24, 2015, 8:05pm

Hi,

On 24.11.2015 20:49, Brian M Hamlin wrote:

Is this an indication that we should pay for experienced MediaWiki consulting here ?

I think, we should disable account creation, and provide an email address (for example this list), for account creation requests.

OR

We run the removeUnusedAccounts.php lets say every 12 hours per cron job, to get rid of the fake accounts.

On a side note, a dev on the SMW list said this:

"

I think the upcoming REL1_26 will also come with the new and improved
Google ReCAPTCHA service

"

BUT if there is someone who says he can fix this sustainably, I would not vote against Mediawiki consulting...

Christian

neteler · December 4, 2015, 4:17pm

Hi,

do we have any plans to avoid that so many spam accounts are opened in our wiki?

Just see
http://wiki.osgeo.org/index.php?title=Special:Log/newusers&offset=&limit=500&type=newusers&user=

Markus

Christian_Willmes · December 4, 2015, 4:21pm

Hi,

nobody answered to the proposal of disabling account creation. I would do this for now.

Also, I am working on the wiki right now, and the server is very slow. I think the server is maybe swapping?

Regards,
Christian

Am 04.12.2015 um 17:17 schrieb Markus Neteler:

Hi,

do we have any plans to avoid that so many spam accounts are opened in our wiki?

Just see
http://wiki.osgeo.org/index.php?title=Special:Log/newusers&offset=&limit=500&type=newusers&user=

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

neteler · December 4, 2015, 4:24pm

On Fri, Dec 4, 2015 at 5:21 PM, Christian Willmes <mail@cwillmes.de> wrote:

Hi,

nobody answered to the proposal of disabling account creation. I would do
this for now.

I thought it was already in place!

Also, I am working on the wiki right now, and the server is very slow. I
think the server is maybe swapping?

Apparently not:

top - 08:22:15 up 435 days, 9:53, 3 users, load average: 10.30, 14.85, 16.18
Tasks: 114 total, 16 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.5%us, 5.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4061456k total, 3154560k used, 906896k free, 170648k buffers
Swap: 4096568k total, 24844k used, 4071724k free, 1359568k cached

(but the load average due to apache is high)

Markus

Christian_Willmes · December 4, 2015, 4:25pm

which means that I propose this (would -> subjunctive, which is the english translation for (german) "Konjuntiv" I think?).
I can do it if people agree though.

Regards,
Christian

Am 04.12.2015 um 17:21 schrieb Christian Willmes:

I would do this for now.

Christian_Willmes · December 4, 2015, 4:29pm

Ok, I disabled account creation for now.

Am 04.12.2015 um 17:24 schrieb Markus Neteler:

On Fri, Dec 4, 2015 at 5:21 PM, Christian Willmes <mail@cwillmes.de> wrote:

Hi,

nobody answered to the proposal of disabling account creation. I would do
this for now.

I thought it was already in place!

Also, I am working on the wiki right now, and the server is very slow. I
think the server is maybe swapping?

Apparently not:

top - 08:22:15 up 435 days, 9:53, 3 users, load average: 10.30, 14.85, 16.18
Tasks: 114 total, 16 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.5%us, 5.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4061456k total, 3154560k used, 906896k free, 170648k buffers
Swap: 4096568k total, 24844k used, 4071724k free, 1359568k cached

(but the load average due to apache is high)

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

neteler · December 4, 2015, 4:32pm

On Fri, Dec 4, 2015 at 5:29 PM, Christian Willmes <mail@cwillmes.de> wrote:

Ok, I disabled account creation for now.

Good - do-ocracy

So I'll trash again the unconfirmed accounts now.

(pro futuro: you may write "if no objections, I'll disable account
creation in x days from now)

Markus

Christian_Willmes · January 13, 2016, 11:40am

Hi,

I did not find a better solution for this spam problem. The captcha solution, that Markus installed and tryed, is what is recommended by the MW community [0].
Should we re-enable user creation on the wiki?
Then we would need to deal with the spam somehow, for example by running the RemoveUnusedAccounts.php [1] via cron (once a day or so)?

Regards,
Christian

[0] https://www.mediawiki.org/wiki/Manual:Combating_spam
[1] https://www.mediawiki.org/wiki/Manual:RemoveUnusedAccounts.php

On 04.12.2015 17:29, Christian Willmes wrote:

Ok, I disabled account creation for now.

Am 04.12.2015 um 17:24 schrieb Markus Neteler:

On Fri, Dec 4, 2015 at 5:21 PM, Christian Willmes <mail@cwillmes.de> wrote:

Hi,

nobody answered to the proposal of disabling account creation. I would do
this for now.

I thought it was already in place!

Also, I am working on the wiki right now, and the server is very slow. I
think the server is maybe swapping?

Apparently not:

top - 08:22:15 up 435 days, 9:53, 3 users, load average: 10.30, 14.85, 16.18
Tasks: 114 total, 16 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.5%us, 5.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4061456k total, 3154560k used, 906896k free, 170648k buffers
Swap: 4096568k total, 24844k used, 4071724k free, 1359568k cached

(but the load average due to apache is high)

Markus
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

neteler · January 13, 2016, 12:04pm

On Jan 13, 2016 12:40 PM, “Christian Willmes” <mail@cwillmes.de> wrote:

Hi,

I did not find a better solution for this spam problem. The captcha solution, that Markus installed and tryed, is what is recommended by the MW community [0].
Should we re-enable user creation on the wiki?

Yes, it is highly needed. Also for the code sprint in Paris, people want to enroll.

Then we would need to deal with the spam somehow, for example by running the RemoveUnusedAccounts.php [1] via cron (once a day or so)?

Yes. Perhaps with a modification of the subscription email that the link expires in 24hs.

Best
Markus

Christian_Willmes · January 13, 2016, 2:39pm

Ok, I re-enabled account creation now.

And I modified the subscription email [1], according to Markus proposal.

Additionally, I uncommented the '$wgChacePage = false;', because the wiki is/was very slow (caused high server load). And this statement was included way back in the config for the old version of the usermap, that is not used anymore now.

Please all keep an eye on the wiki, if something does not work anymore due to the caching change.

Thanks,
Christian

[1] https://wiki.osgeo.org/wiki/MediaWiki:Createaccount-text

On 13.01.2016 13:04, Markus Neteler wrote:

On Jan 13, 2016 12:40 PM, "Christian Willmes" <mail@cwillmes.de <mailto:mail@cwillmes.de>> wrote:
>
> Hi,
>
> I did not find a better solution for this spam problem. The captcha solution, that Markus installed and tryed, is what is recommended by the MW community [0].
> Should we re-enable user creation on the wiki?

Yes, it is highly needed. Also for the code sprint in Paris, people want to enroll.

> Then we would need to deal with the spam somehow, for example by running the RemoveUnusedAccounts.php [1] via cron (once a day or so)?

Yes. Perhaps with a modification of the subscription email that the link expires in 24hs.

Best
Markus

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

Christian_Willmes · January 13, 2016, 2:46pm

Hi,

can someone please help in setting up the cron job? I guess I don't have the permissions.

The following command should be run once a day:

'php /var/www/wiki_osgeo/maintenance/removeUnusedAccounts.php'

Thanks,
Christian

On 13.01.2016 15:39, Christian Willmes wrote:

Ok, I re-enabled account creation now.

And I modified the subscription email [1], according to Markus proposal.

Additionally, I uncommented the '$wgChacePage = false;', because the wiki is/was very slow (caused high server load). And this statement was included way back in the config for the old version of the usermap, that is not used anymore now.

Please all keep an eye on the wiki, if something does not work anymore due to the caching change.

Thanks,
Christian

[1] https://wiki.osgeo.org/wiki/MediaWiki:Createaccount-text

On 13.01.2016 13:04, Markus Neteler wrote:

On Jan 13, 2016 12:40 PM, "Christian Willmes" <mail@cwillmes.de <mailto:mail@cwillmes.de>> wrote:
>
> Hi,
>
> I did not find a better solution for this spam problem. The captcha solution, that Markus installed and tryed, is what is recommended by the MW community [0].
> Should we re-enable user creation on the wiki?

Yes, it is highly needed. Also for the code sprint in Paris, people want to enroll.

> Then we would need to deal with the spam somehow, for example by running the RemoveUnusedAccounts.php [1] via cron (once a day or so)?

Yes. Perhaps with a modification of the subscription email that the link expires in 24hs.

Best
Markus

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac

neteler · January 22, 2016, 11:05am

On Wed, Jan 13, 2016 at 3:46 PM, Christian Willmes <mail@cwillmes.de> wrote:

Hi,

can someone please help in setting up the cron job? I guess I don’t have the
permissions.

The following command should be run once a day:

‘php /var/www/wiki_osgeo/maintenance/removeUnusedAccounts.php’

Done:

wiki:~/cronjobs# crontab $HOME/cronjobs/cron_job_list && crontab -l

add jobs here and then run:

crontab $HOME/cronjobs/cron_job_list && crontab -l

±--------------- minute (0 - 59)

| ±------------ hour (0 - 23)

| | ±--------- day of month (1 - 31)

| | | ±------ month (1 - 12)

| | | | ±— day of week (0 - 6) (Sunday=0 or 7)

| | | | |

* * * * * command to be executed

daily unused accounts cleanup

05 22 * * * (cd /var/www/wiki_osgeo ; php maintenance/removeUnusedAccounts.php --ignore-groups emailconfirmed --delete )

Markus

Christian_Willmes · January 22, 2016, 12:23pm

Thank you!

Am 22.01.2016 um 12:05 schrieb Markus Neteler:

On Wed, Jan 13, 2016 at 3:46 PM, Christian Willmes <mail@cwillmes.de <mailto:mail@cwillmes.de>> wrote:
> Hi,
>
> can someone please help in setting up the cron job? I guess I don't have the
> permissions.
>
> The following command should be run once a day:
>
> 'php /var/www/wiki_osgeo/maintenance/removeUnusedAccounts.php'

Done:

wiki:~/cronjobs# crontab $HOME/cronjobs/cron_job_list && crontab -l
# add jobs here and then run:
# crontab $HOME/cronjobs/cron_job_list && crontab -l

# +---------------- minute (0 - 59)
# | +------------- hour (0 - 23)
# | | +---------- day of month (1 - 31)
# | | | +------- month (1 - 12)
# | | | | +---- day of week (0 - 6) (Sunday=0 or 7)
# | | | | |
# * * * * * command to be executed
#
# daily unused accounts cleanup
05 22 * * * (cd /var/www/wiki_osgeo ; php maintenance/removeUnusedAccounts.php --ignore-groups emailconfirmed --delete )

Markus

_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/sac