You are not logged in.

#1 2019-05-17 2:20 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Running a Diary being hit by bots without StopForumSpam

In a nutshell, you get swamped with spam
OpenStreetMap Diaries

(plus all your own Diary posts with spam-stats or spam-advice get summarily removed and you get threatened with being tarred & feathered (I made that last bit up, but when the head of the Data Working Group gets told by me "please do not send me any more emails" & sends one anyway), well try this for surreal 1984 stuff:-

OSM DWG wrote:

Everybody else in OSM is allowed to make snarky, cynical, or sarcastic comments … the rules that apply to you specifically have been tightened

Even if everything you have said was factually correct, and everything everyone else said was factually wrong, this would not change our judgement one bit

At this moment, the first 9 pages (20 posts a page) of the OpenStreetMap (OSM) diary are full of nothing other than Chinese-language spam (as best as I can tell, advertising Guangzhou hotel services such as Saunas; personal services, I think they call it). [Update: an hour later a mod had removed them all. 22:08 BST: it is back up to p6; at 01:00 BST the wfgz spammers will startup again and at 07:50 BST a min 14 pages were removed.] As a 19-year veteran of fighting spam I wrote some step-by-step advice (post deleted by OSM DWG) on how to fix it, which was firmly ignored.

Worldwide spam soared on May 12 and had doubled by May 14. This is most unusual as Peak Season for spam is normally October-December. One interesting feature is that the events inside OSM seem a precursor to that.

On April 22 spam began to rise on OSM (post deleted by OSM DWG) hitting 30,000 daily spam-posts on April 28. These folks were posting in Bengali (BN)† and every post was terminated with the ASCII letters “wfgz”. I tracked that spam using a 10-minute cron-job during the wee hours of May 14 UTC (see bottom of this post (post deleted by OSM DWG) as to find the rate at which these folks were posting (max. 16/minute). An OSM Mod reported on May 10 (post deleted by OSM DWG) that 10,000 blocked accounts contained ~3,500 unique IPs (~3 accounts/IP). That Mod thought that he was dealing with a bot-net; he did not realise that this was simply normal spam experience. Sadly, SFS is not used by OSM and therefore not a single one of these IPs, email-addresses nor usernames have been submitted to SFS by them.

†Strangely the spammers were posting in Chinese. Somehow the OSM system consistently registered them as bn Bengali. I have no idea why.

franchise image

Spam facts:

  1. 95% of all spam is due to spam-bots provided by folks such as XRumer (the rest is posted directly by humans)

  2. Spam-bots are a commercial Franchise (exactly like McDonalds)

  3. The central business provides the software + business system

  4. The franchisee provides the computer, internet connection + time

The spam-bot is largely an exercise in artificial intelligence, teaching computers to view human beings & their work as Prey. Previously, the bots were specialised for particular open-source forums, such as this one. Before today that did NOT include OSM Diaries. However, since late April 2019 it DOES include OSM and, I suspect, rather a lot of other forum software as well.

Offline

#2 2019-05-17 5:25 pm

kpatz
Member
Registered: 2008-10-09
Posts: 1,437

Re: Running a Diary being hit by bots without StopForumSpam

Whoever is behind the scourge of Xrumer needs to have a visit from two big guys with bats.


Spam happens when greed meets stupidity.

Offline

#3 2019-05-17 6:24 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

You need to understand something to defeat it. So, I held my nose, put the Hazmat suit on, and gathered some intelligence from the source:

800px-FlashSuit.jpg

Botmaster are currently offering 4 products to license:-

  • XRumer 18.0.1 (2019-01-26)

This software will help to increase traffic to website to hundreds, thousands times. Program have a rich seven year history, which use experience of professionals in search engine optimization. Appreciate and use a truly unique and powerful XRumer program, can both professionals and beginners.

  • XEvil 3.0.4.123

it’s a new revolutionary tool included in XRumer SEO package since January, 2017 (available only for Standard and Business license)

XEvil allow to automatically recognize more than 8400 types of captchas with a very high recognition speed — more than 100 images per second. New technology of decoding allow extremely fast and precisely to decode text, numbers, arithmetic's and symbols even on hard types of captchas, no matter of their size, noise, deformations and font type. No matter how hard is captcha, speed of decoding it’s always same fast (~0.01 sec per 1 captcha).

ATTENTION: ReCaptcha-2 Module (XEvil 4.0) is in final stage of beta testing. At the moment, it requires a lot of resources. For stable usage of software it's recommended to run on multiprocessor servers with 32 GB RAM or more.

  • Hrefer 5.0.5 Professional

Helper program Hrefer — is the program created to search for the new links to forums, guestbooks, blogs, Wiki, etc. that can then be used as a target list for the main XRumer application. This software is used to automatically parse results from most popular Search Engines including Google, Yahoo, Bing and Yandex, with highly diversified queries for maximum efficiency.

ATTENTION: Hrefer comes for FREE at XRumer purchase.

  • SocPlugin 4.0.68

A new tool that works with most popular social networks: posting, inviting, commenting, liking, autofilling, and a lot of more important functions.

ATTENTION: SocPlugin comes for FREE at XRumer purchase (except for a Lite-version).

Note: a “BlogsPlugin” is mentioned in the Title but not within the body of the Home page. Evidently that is still coming.

You may also be interested to read the history of XRumer (hxxp://www.botmasterlabs.net/events/):

XRumer 18.0.1 + SocPlugin 4.0.68 (2019-01-26)

Important update of XRumer, in which was significantly improved logic of profile registration on many engines. It was improved work with Bitrix, Joomla, WordPress Forum, MyBB, VBulletin, XenForo; It was added mechanism of text changing depending of recipient site topic (new macros #theme); were updated default databases and now they contain more that 8 million link, Improved work with HTTPS and Google ReCapctha-2 and many other important improvements…

XRumer 16.0.18 + SocPlugin 4.0.63 (2018-09-23):

Databases that comes with software were updated and checked. Total size was increased to 8 million links of supported recourses such as blogs, forums, guestbooks, boards, BBS, CMS and other platforms. Database of known textcaptcha was increased with more than 2000 new answers to antibot-questions and now are contains more than 324000 textual captcha. Significantly increased stability and speed of work. Same was optimized usage of PC recourses: maximal number of threads can be set up to 1000 and more (all depends of usage mode). It was improved work with HTTPS. Highly increased success rate and quality of sending PM (personal messages). Same as many other improvements and bug fixies.

XRumer 16.0.16 + SocPlugin 4.0.61 (2018-05-21):

Important updated with significantly increase of success rate in decoding of Google ReCaptcha (taking in consideration that ReCaptcha v1 was closed on march 31). Increased efficiency of work with new version of XenForo, IPBoard, VBulletin, phpBB, added few new engines, improved work with Unicode (UTF-8) for multi language uses. Significantly improved logic of "Antisзam" feature. In SocPlugin was updated work with Facebook and VK. And 70+ other improvements and fixes…

XRumer 16.0.15 + SocPlugin 4.0.56 (2018-01-26)
XRumer 16.0.14 + XEvil 3.0.4 (2017-10-04)
XRumer 16.0.11 + Hrefer 5.0.2 (2017-07-20)
XRumer 16.0.8 + XEvil 3.0 (2017-05-01)
XRumer 16.0 + XEvil (2017-01-03)
XRumer 12.0.18 — 12.0.19 (2016-10-29)
XRumer 12.0.17 (2016-08-10)
XRumer 12.0.16 (2016-02-29)
SocPlugin 4.0.31 — 4.0.33 (2016-02-12)
XRumer 12.0.12 (2015-10-12)
XRumer 12.0.11 + SocPlugin 4.0.22 + New contest! (2015-05-27)
XRumer 12.0.9 and SocPlugin 4.0.16 (2015-12-15)
XRumer 12.0.8 and SocPlugin 4.0.15 (2015-11-30)
XRumer 12.0.7 Elite (2014-09-12)
XRumer 12.0.6 Elite (2014-03-15)
XRumer 12.0 Elite (2013-11-21)
XRumer 7.7.45 Elite [fix] (2013-05-05)
XRumer 7.7.42 Elite (2013-03-16)
XRumer 7.7.41 Elite (2013-01-31)
XRumer 7.7.40a Elite (2012-12-31)
XRumer 7.7.35 Elite (2012-11-16)
XRumer 7.5.31 Elite (2012-10-16)
XRumer 7.5.30 Elite (2012-09-14)
XRumer 7.5.28 Elite  [Beta] (2012-08-25)
XRumer 7.5.28 Elite (2012-08-12)

Offline

#4 2019-05-17 8:13 pm

kpatz
Member
Registered: 2008-10-09
Posts: 1,437

Re: Running a Diary being hit by bots without StopForumSpam

Cracking captchas with bots needs to be made illegal under penalty of death.

There's no reason to crack them except to perform acts of evil we don't want, which is why we have captchas in the first place.  (Ok, maybe for sight impaired individuals... but most captcha systems offer other options for this).

XEvil might be the most aptly named piece of software in the universe.

MALWARE: "Software that is written and distributed for malicious purposes, such as impairing or destroying computer systems."

Xrumer is malware by this definition, it is used for malicious purposes, as in breaking into and spamming forums and blogs.  Distributing malware is illegal in most countries.  The Xrumer people need to to to JAIL, plain and simple.  Federal pound-you-in-the-A$$ prison, for a long, long, long, long time.

Last edited by kpatz (2019-05-17 8:20 pm)


Spam happens when greed meets stupidity.

Offline

#5 2019-05-17 9:04 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

I think that most forum webmasters will be astonished that XRumer is distributed so openly. No Dark Web nor Tor browser is required for this. Stick 'xrumer' in the search box & the distributor will be the #1 result. Their competitors are probably making use of Google Adwords.

Offline

#6 2019-05-18 11:53 am

Oblivian
Member
Registered: 2018-11-04
Posts: 79

Re: Running a Diary being hit by bots without StopForumSpam

Likewise, search 'india freelance SEO (or paid link)' And look at all the individuals and companies openly advertising themselves as able to get your site heaps of links and google ranks. Then guess the tools they use and how they do it...

A few pages in you find the ones where the guys at home don't use bots but by hand use the lists and go and setup profiles and add data to them 4 or 5 at a time (you can visibly see the delays in registering, confirming registration and then actually logging in to edit). Paid in the region of $5us per 100 links. or $3-5/hr. advertising as article or blog writers.

Last edited by Oblivian (2019-05-18 12:13 pm)

Offline

#7 2019-05-19 3:39 am

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

BlogsPlugin 1.0.5 comes for free with all Standard and Business license XRumer 12.0

BlogsPlugin is working with 6 most popular engines:

  1. Wordpress

  2. Blogger

  3. Livejournal

  4. Tumblr

  5. Bloglines

  6. Netvibes

Offline

#8 2019-05-19 3:57 am

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

Improvements in XRumer 12.0.11 (2015-05-27):

  • trained to new type of graphical captcha. Also, was significantly improved decoding of new ReCaptcha

  • added new databases due to improvement of work with engines, the total was got more than 300,000 unique resources:

  • added new semi-manual mode to enter textual captcha.

  • Mass-PM system was significantly updated |
    Added support of XenForo. This engine is very valuable because it does not have any captcha at PM sending and does not have any limits on number of messages sent |
    Also added in Mass-PM support of several versions of Discuz |
    added support of MyBB engines |
    Improved work of Mass-PM with forums on phpBB, SMF

  • increased the success rate and improved work with several engines:
    Discuz
    XenForo
    MyBB
    Bitrix
    DLE

  • Very many other improvements, including...

  • Updated algorithm to bypass CloudFlare protection

  • increased percentage of successful registration on latest XenForo versions

  • added special pauses at registration on MyBB recourses to bypass the time control at form filling

  • adjusted the processing of CloudFlare (applied a new mechanism of decoding of ReCaptcha)

Offline

#9 2019-05-19 5:17 am

Oblivian
Member
Registered: 2018-11-04
Posts: 79

Re: Running a Diary being hit by bots without StopForumSpam

I don't know how they are still peddling it as a valid SEO tool!

Everyone knows that google down-ranks you for fake links since the last couple of AI updates. And a good site has Nofollow specified anyway. All it does is peeve admins/moderators off and down-grade the site they target too

Wikipedias page on Google_penalty is pretty clear cut that it's basically breaching all their terms and ranking

Offline

#10 2019-05-19 8:01 am

kpatz
Member
Registered: 2008-10-09
Posts: 1,437

Re: Running a Diary being hit by bots without StopForumSpam

I don't know how they are still peddling it as a valid SEO tool!

I don't know how they can sleep at night.

All the forum and blogging software companies NEED to start following Xrumer's release notes closely and quickly develop countermeasures.  Such as:

Added support of XenForo. This engine is very valuable because it does not have any captcha at PM sending and does not have any limits on number of messages sent

Xenforo needs to respond immediately with added protection in their PM system, for example.  (Granted, PM spam is about as stupid as it gets..no SEO benefits whatsoever, and only people as stupid as spammers will fall for something spammed to them via PM anyway)

If forum and blog software companies started taking spam more seriously, they could put a big dent in the problem.  Captcha makers, Cloudflare too.  Find better ways to make them unbreakable, or much more difficult to break.  They should update as soon as they're cracked.  They should develop new approaches ahead of time so they can be deployed quickly.

It's like a war out there, and with a few exceptions (like SFS), the bad guys are winning.

Maybe I should start my own company, developing new forum and blog software that puts anti-spam front and center.

Last edited by kpatz (2019-05-19 8:10 am)


Spam happens when greed meets stupidity.

Offline

#11 2019-05-19 4:29 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

I think that you are pointing at the wrong people.

If you look at hxxp://www.botmasterlabs.net/buy1/#cards ('Purchase XRumer + XEvil + Hrefer + SocPlugin') you will see the following payment options for those wanting to licence XRumer & it's hell-hounds:-

  • VISA / MASTERCARD (and others)

  • Alipay

  • PayPal

  • WebMoney

  • BitCoin

  • MoneyGram

  • Western Union

  • Wire Transfer

A class-action in the USA should be able to transfer 7 or 8-figure $$$ amounts from some of those folks. A tap on their systems should also be able to liberate the names+banks of those licensing the product, and just a little bit of work will show criminal action against several hundred thousand sites each from thos folks, which should release very large volumes of money indeed into the pockets of the governments (need to give govt a reason to act).

It has also occurred to me that a little hacking of XRumer should be able to feed back it's activity. But of course, that would be a criminal act, which I cannot condone.

With not a lot of thought it will be easy indeed to grab them by the nuts & turn them upside down to grab their lucre.

Offline

#12 2019-05-23 6:22 am

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

I've integrated much of the information that I've discovered about XRumer & it's cohorts here to give a more-coherent timeline & description in my OSM post:

OSM is now within an iteration of spam-bot software (such as XRumer)
(post deleted by OSM DWG June 24; content below)

Most of the time currently (07:50 BST 18 May 2019) these Diary posts are lost within floods of Chinese (ZH-CH) or Bengali (BN) spam. However, a hard-pressed Mod has just removed the latest torrents of ordure poured over these diaries (I had just reached p15 in my search for solid ground when it all disappeared) so, for a short while, this post will be easy to see.

It seemed to me that that this event in OSM was turning out to be a classic depiction of how NOT to handle spam so began to scribble about it within StopForumSpam (SFS) for the benefit of the wider world. That led into an examination of the BotMaster Labs’ product used by their Franchisees (it is licensed rather than sold): XRumer.

The Spam-Spreader Bundle from Botmaster Labs

Almost all spam is spread by bots. That is as true for forums, guestbooks, blogs, Wiki & Social Media as it is for Email, but for all except email one of the primary s(pam|hit)-spreaders is XRumer (Wikipedia) from Botmaster Labs, with just a very little spam spread directly by human hand.

XRumer was first published on 12 August 2012 wrote:

This software will help to increase traffic to website to hundreds, thousands times.

The base software automates account creation, confirmation, login & post for as many different websites and as fast as the Licensee’s computer & network can handle. The important thing to understand about XRumer & all of it’s helper programs is that everything is scripted, everything is automated and, once set up & customised by the Franchisee, it will work 24/7/365.

The principle helper program is Hrefer wrote:

Helper program Hrefer — is the program created to search for the new links to forums, guestbooks, blogs, Wiki, etc. that can then be used as a target list for the main XRumer application. This software is used to automatically parse results from most popular Search Engines including Google, Yahoo, Bing and Yandex, with highly diversified queries for maximum efficiency.

ATTENTION: Hrefer comes for FREE at XRumer purchase.

Hrefer’s action is automated, as is the interaction between it & XRumer. Both rely on the fact that almost all forum packages are Open Source & that therefore their source-code is published & well-known. Hrefer drills down not just to the package publisher but also to the package-version & customises it’s routines towards both as to automate every aspect of gathering lists of forums, etc. (Hrefer) and then the mechanics of actually spamming them (XRumer).

BlogsPlugin began to be provided with XRumer 12.0 (21 Nov 2013) wrote:

BlogsPlugin is working with 6 most popular engines:

  1. Wordpress

  2. Blogger

  3. Livejournal

  4. Tumblr

  5. Bloglines

  6. Netvibes

SocPlugin began to be provided with XRumer 12.0.8 (30 Nov 2015) wrote:

A new tool that works with most popular social networks: posting, inviting, commenting, liking, autofilling, and a lot of more important functions.

ATTENTION: SocPlugin comes for FREE at XRumer purchase (except for a Lite-version).

XEvil began to be provided with XRumer 16.0 (3 Jan 2017) wrote:

it’s a new revolutionary tool included in XRumer SEO package since January, 2017 (available only for Standard and Business license)

One of the ways that forum authors began to respond to folks like Botmaster Labs was by introducing Captcha (Wikimedia), inverting the Turing Test by providing a way for machines to tell a human from a machine. XEvil is designed to circumvent that test:

XEvil allow to automatically recognize more than 8400 types of captchas with a very high recognition speed — more than 100 images per second. New technology of decoding allow extremely fast and precisely to decode text, numbers, arithmetic’s and symbols even on hard types of captchas, no matter of their size, noise, deformations and font type. No matter how hard is captcha, speed of decoding it’s always same fast (~0.01 sec per 1 captcha).

ATTENTION: ReCaptcha-2 Module (XEvil 4.0) is in final stage of beta testing. At the moment, it requires a lot of resources. For stable usage of software it’s recommended to run on multiprocessor servers with 32 GB RAM or more.

XRumer 18.0.1 is an across-the-board improvement of all parts (26 Jan 2019) wrote:

Important update of XRumer, in which was significantly improved logic of profile registration on many engines. It was improved work with Bitrix, Joomla, WordPress Forum, MyBB, VBulletin, XenForo; It was added mechanism of text changing depending of recipient site topic (new macros #theme); were updated default databases and now they contain more that 8 million link, Improved work with HTTPS and Google ReCapctha-2 and many other important improvements…

The last 3 iterations of this package (it comes bundled together with XEvil, Hrefer Professional and SocPlugin, although the latter is NOT provided with XRumer Lite) are as follows:–

  • 2018-05-21: XRumer 16.0.16 + SocPlugin 4.0.61

  • 2018-09-23: XRumer 16.0.18 + SocPlugin 4.0.63

  • 2019-01-26: XRumer 18.0.1 + SocPlugin 4.0.68

You will therefore understand that BotMaster have been updating their products recently on a 3-a-year basis, and that 2018’s first update was at this time of the year.

StopForumSpam fights Fire with Fire

StopForumSpam (SFS) provides an API which allows auto-detection of known spammers trying to register a Profile within your Forum/Guestbooks/Blog/Wiki/Social-Media. Our current (and historic) stats indicate that ⅓ of all email queries to the API are positive for such spammers. More than 200,000 webmasters make use of the SFS API.

SFS runs a large number of Honeypot sites that feed current spammers into the SFS database. We also rely on Webmasters sending reports of spammer details to SFS. In this way it is a community effort in which all notify each other of current threats.

The evidence from SFS is that worldwide-spam this year surged on 26 April (+33% in 2 days), 9 May (+51% in 1 day) and 15 May (+101% in 2 days). Correlating these figures with OSM we see that (post deleted by OSM DWG) this recent bot-spam took off on 25 April, hit 5 figures on 9 May and again on the 17 May.

There isn’t a shred of a doubt in my mind that OSM is being hit by a bot-net such as that operated by XRumer, possibly in the next iteration (it normally gets upgraded in May). However, I’m also certain that the OSM Mods are living in dreamland, and that these latest spam are just the opening salvos of exploratory forays. The main advance is yet to come. If my tentative adding up is correct then these ZH-CN & BN spam are just testers of a latest spambot iteration. Just wait until it is officially released & the whole world upgrades. Something else to look forward to.

Contra-Indications 28 May

23 May (post deleted by OSM DWG) was the last major spam-attack (3,682 spam posts in a single day) and then back to the no-edit single-post spammers that are the recent background to these diaries. It seems like the bot-wave attacks may have ceased. [Update 9 June: that was because the OSM admin have killed OSM Diaries dead (post deleted by OSM DWG)]

I agree with the OSM Admin in one respect; it makes zero sense for XRumer (or any other similar pests) to include OSM within their routines. Each person using XRumer relies on hitting 1000s of similar forums each night. They do not care what happens to any individual forum nor whether each individual attack is successful or not. It is a numbers game, and they only care whether Google sees their post — nothing else. They will get paid if Google records enough spam posts by their bot. OSM only gives them one shot, and that is not enough. They want 1000s of shots.

May is normally the bottom of the spam Year, and the period leading up to Xmas the top. And yet, on 23 May SFS recorded a 3-fold increase in API queries. And then the OSM bot-wave stopped. Who knows if the two are connected!

The XRumer website still does not show any new version. I do not have any clue what is happening, but well done to the OSM admin as they seem to have weathered the current storm. Let us hope that I am wrong, and that it does not spread as a concerted effort to the map.

Another thread has been opened by a different user on updates to xRumer.

Offline

#13 2019-05-23 12:09 pm

kpatz
Member
Registered: 2008-10-09
Posts: 1,437

Re: Running a Diary being hit by bots without StopForumSpam

Just read that thread... and some of them over there are short sighted as I'm sure you're aware.  I'd love to add to it but I don't want to register over there just to make 1-2 posts.

If they don't want to require map edits before posting to the diary (which would just give spammers incentive to attack the maps even sooner), then implement a moderation queue on the diary for all new posters, where all new posts, especially those with links, must be moderated before making them visible.

Someone needs to take action against the Xrumer folks in any case.  I don't know what country they're based out of, but they're breaking multiple laws in several countries at the very least, and they must pay for their actions.  If lawyers weren't so bleeping expensive, it would be easier.


Spam happens when greed meets stupidity.

Offline

#14 2019-05-23 1:23 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

kpatz wrote:

Someone needs to take action against the Xrumer folks in any case

They have been operating without interference since 2012.

Possibly SocPlugin (Social Media bot, details above) may be the way to get the authorities to take notice. I kept a careful eye on the 2016 furore & connection between Russian bots & the USA election, yet never heard a whisper about Botmaster Labs nor SocPlugin.

Offline

#15 2019-05-23 3:32 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

kpatz wrote:

If they don't want to require map edits before posting to the diary … then implement a moderation queue on the diary for all new posters, where all new posts, especially those with links, must be moderated before making them visible.

Essentially that is exactly what I've suggested.

Posts containing links from new users already have the links auto-removed.

Creating an a/c on OSM currently immediately gives the right to both edit the map and to make Diary posts/comments (and also a few other esoteric privileges related to the map). My most succinct suggestions to stop the spam-storm are:–

  1. That the diary POST privilege depends upon someone also being a mapper (no map edits, no post)

  2. That new mappers are placed into a (hidden) moderation queue to ensure that they are human

At this moment 6% of all diary posts are from non-mappers. In addition, many of those posts are nonsensical. Try this one for size. It conists of 2 characters:–

  • “…” (U+2026 HORIZONTAL ELLIPSIS)

  • “.” (U+002E FULL STOP)

… and that is it.

The admin refuses to remove it even though he has been informed of the user and URL.

Offline

#16 2019-05-23 8:56 pm

sklerder
Member
Registered: 2012-10-11
Posts: 336
Website

Re: Running a Diary being hit by bots without StopForumSpam

Hi !

Well, if a bot can automatize registration and posting, it's because the registration form gives all the indications to easily fill the form smile
If the field for email is not named "email" or doesn't contain this word, it becomes harder to automatize.
Better, if the field name email is present in the form, but not visible for humans (by CSS property, for example), it shouldn't be filled by humans (because they don't see it), but will probably filled by bots.
Then, checking the content of the fake field should help to detect a bot wink
That's the way it's done with "SpamBarrier" and "StopForumSpam+Honeypot" mods on FluxBB, and Papa Parrot can confirm it works ...

Offline

#17 2019-05-23 10:20 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

XRumer is auto-customised to specific software, and also to different versions within that software. Therefore, to disrupt the bots you need to significantly change your distro's setup from what the bots will expect.

That is why I suggested changing the OSM setup; at the moment every newly-created user can both Edit & Post. Moderating new users would immediately block all bot-actions.

One feature that I found interesting was clear indications within Botmaster's description that they are making their best efforts to introduce Artificial Intelligence (AI) into their setup, so that the software auto-reconfigures itself according to user actions.

Offline

#18 2019-05-24 11:25 am

kpatz
Member
Registered: 2008-10-09
Posts: 1,437

Re: Running a Diary being hit by bots without StopForumSpam

Another fix, for the nonsensical "one character" posts would be to enforce a minimum post length.  Many forums don't allow 1 character posts.

AI?  Geez... the botware will be smarter than the spammers then.  I've always thought forum software should have a morph feature where it renders the HTML for registration and posting forms differently every time... make it look the same for the user, but the order, name, and positions of each field change on every call, and the names are random (so the bot can't find the "email" field by name).  The bot would have to parse the HTML, the CSS, and attempt to match up the user readable titles to the actual fields to fill them in properly.  Put in a bunch of hidden dummy fields, and the odds of a successful bot hit goes down dramatically.

Using Javascript will help too... I don't think most bots execute Javascript.  Make the page render in script, and have script sanitize the data, and put key fields in that aren't filled by the user that the server checks, and the bot won't know how to fill them in... unless it does execute Javascript... and then, we could slow them down by making the script run a calculation that takes 5-10 seconds on a modern PC and passes the result back to the server for validation.  On a regular browser, the calculation could run while the user is filling out the registration form, but a bot would have to either run the calculation itself, slowing it down, or skip it and be rejected.

Last edited by kpatz (2019-05-24 11:28 am)


Spam happens when greed meets stupidity.

Offline

#19 2019-05-24 10:10 pm

sklerder
Member
Registered: 2012-10-11
Posts: 336
Website

Re: Running a Diary being hit by bots without StopForumSpam

Hello !

When I wrote SpamBarrier, and particularly the HoneyPot part, I implemented it with a modifiable field name to replace the email field of the form.
The simple fact of renaming it, an keeping a fake email form field, six years later, do actually work in really more than 999/1000 cases (I could multiply by ten) ...
It's just a proof of concept, it was was not my idea, but it's really an efficient way to fear those idiot bots smile

Last edited by sklerder (2019-05-24 11:12 pm)

Offline

#20 2019-05-25 9:38 am

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

kpatz wrote:

I don't know what country they're based out of

Investigate the site CSS & you will find constant references to Russian (RU) as the language. Since Ukraine has been the permanent top of SFS's country-chart, the best guess is the breakaway Russian-speaking part of eastern Ukraine. Y'know, the one that in 2014 shot down the Dutch civil airliner at 30,000 feet with a Russian military Buk missile. That one.

Offline

#21 2019-05-30 5:34 am

masterwolf
Member
From: 123 North Antarctica
Registered: 2012-01-21
Posts: 54
Website

Re: Running a Diary being hit by bots without StopForumSpam

Oh neat. I was going to post OSM's spam problem here but I see we have a mapper for a mod. The first time I the dairy spam, the bots were adding 1-3 posts per second. I stop following the spam after 200 pages (4000 posts).

Offline

#22 2019-05-30 12:47 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

masterwolf wrote:

(diary spam) the bots were adding 1-3 posts per second

I've added a paragraph at the bottom of the 18 May post:

OSM post about XRumer wrote:

I agree with the OSM Admin in one respect; it makes zero sense for XRumer (or any other similar pests) to include OSM within their routines. Each person using XRumer relies on hitting 1000s of similar forums each night. They do not care what happens to any individual forum nor whether each individual attack is successful or not. It is a numbers game, and they only care whether Google sees their post — nothing else. They will get paid if Google records enough spam posts by their bot. OSM only gives them one shot, and that is not enough. They want 1000s of shots.

Your observation matches my own measurements. It was definitely Bots that were posting spam into OSM. But that makes no sense! The only thing that may make sense is that this was all a practice run for mass-spamming the map itself.

Offline

#23 2019-05-30 2:15 pm

Papa Parrot
Member
From: Mexico
Registered: 2011-08-19
Posts: 1,826
Website

Re: Running a Diary being hit by bots without StopForumSpam

I am wondering what kind of software this "diary forum" is using, and why no one makes a script, similar to what sklerder uses in Fluxbb, I use that my self, and get 0 bots, successfully registering and posting, No Xrunner, or others. I suppose if the software being used is closed source, or exceedingly complex, like a "plate of spaghetti", (phpBB), then it might not be possible or very easy to modify and use that, ...but it makes zero sense to me that any forum would permit bots to register when it is not that hard to detect and block them.

Alex Kemp>The admin refuses to remove it even though he has been informed of the user and URL

It sounds to me like the site owner/admin wants the site to be spammed, all though I can not understand why ,
other wise they would be more supportive, and want things like that removed.
  One thing I have noticed on other forums, once the bots, and /or the persons behind the bots, once they start getting easy access, more and more start coming, and will keep coming, .....

Offline

#24 2019-05-30 3:06 pm

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

Papa Parrot wrote:

I am wondering what kind of software this "diary forum" is using

It is open source, stored on github. The 'website' is a vast conflation of the OSM Map, Users, API, Diary, Forum, Comments, Wiki, gaah, it goes on & on … If you think that SFS handles lots of business you should try OpenStreetMap. Scores of servers worldwide.

https://github.com/openstreetmap/openst … site/pulls
https://github.com/openstreetmap/chef/pulls
https://munin.openstreetmap.org/

If you examine munin for the website (www) you will see that Apache is handling upto 250 connections per second across Port-80. For comparison, SFS currently handles ~177 API queries/second. Those OSM connections handle 2M bytes / sec, which I believe is a bit more than SFS (and one should hope so, seeing as SFS handles it's load with just one machine whilst OSM has 90 servers to handle it's load).

Offline

#25 2019-06-07 10:15 am

Alex Kemp
Moderator
From: Nottingham, England
Registered: 2009-12-02
Posts: 2,420
Website

Re: Running a Diary being hit by bots without StopForumSpam

Update on the OSM spam problem:
The OSM Admin have fixed the issue of spam flooding the OSM diaries. They have achieved that by ending supply of all Diary pages to bots worldwide. Essentially, they have killed the disease by killing the patient. I am still in shock at the attitude of the Admin.

This is the current (Thu, 06 Jun 2019 17:07:41 GMT) osm.org/robots.txt:

$ wget -S osm.org/robots.txt
…
$ cat robots.txt
User-agent: *
Disallow: /user/*/diary
Disallow: /user/*/traces/
Allow: /user/
Disallow: /traces/tag/
Disallow: /traces/page/
Disallow: /api/
Disallow: /edit
Disallow: /browse
Disallow: /diary
Disallow: /login
Disallow: /geocoder
Disallow: /history
Disallow: /message
Disallow: /trace/
Disallow: /*lat=
Disallow: /*node=
Disallow: /*way=
Disallow: /*relation=

Host: www.openstreetmap.org

A word of warning for novice webmasters: this is a dangerous file to mess with. Get it wrong and you can de-list your entire site from all Search Engines.

It is the first 2 lines that have been recently added to (successfully) de-list all Diary pages and therefore stopped the spam dead in it's track. I was stunned to discover this setup, as the last four lines also de-list the OSM Map from all Search Engines†. As best as I can tell, those last lines have been there since at least 2009 and mean that the OSM map is a closed gulag restricted to just one site (osm.org) and cannot be searched by any other means†.

For more info see Extreme SEO on osm.org kills both Spam & OSM (post deleted by OSM DWG).

† My error here. The last 4 lines are to prevent archaic methods of addressing the map from years ago that are no longer used. So why were large parts of the map missing (I checked most carefully)? The answer is collateral damage from changing robots.txt. I've met that sequence many times; you should not mess with that file if you can help it.

Offline

Board footer

Powered by FluxBB

Close
Close