A Strategy for Discouraging SPAM

Zenkakuji · 2007-12-25 8:18 pm

Hello Everyone,

I am relatively new to administering an online forum, and have been dealing with inappropriate spam on my site. It got so bad, we decided to deactivate the forums while coming up with an approach to stop all of the unwanted posts.

We just started using this strategy for about 2 months and have seen some successful results with a 95% reduction in spam postings.

The following steps were employed to get a handle on minimizing spam postings:

1. Deactivate forums on site. Place a note to users that the forum is currently inactive and will resume in the near future without a specific date if possible. Make the forums not readable by the general users.

2. Rather than delete the spam postings, utilize the content to discourage future similar posts.

The "phpBB" forum toolset contains a "Word Censor" feature. This allows the administrator to enter words that are automatically translated when posted & viewed on the website.

As an example, let us suppose that you do not want the word "shoe" or any word starting with "shoe" to be visible on your website. In the Word Censor administrative tool, add the word 'shoe*' without the single quotes, and translate it to '=' or some other single character.

You may want to enter other possible mis-spellings of "shoe" such as 'shue*', or 'shu*' and translate these to the same single character.

After entering in a number of word translations, preview the posted inappropriate forum to confirm that the translation is eliminating words & phrases as desired.

For our website, we have a list of offensive words and mis-spellings noted from spam postings. In addition, we have entered a number of pharmaceudical product names that have been spammed on our site as well.

3. Many spammers are posting URLs to increase the posting metrics for search engines. If you notice that you are receiving a lot of unwanted website URL postings, consider adding word translations for portions of the URLs.

Example: to eliminate the viewing of a website from a Russian server,
enter '*.ru*' into the Word Censor tool and translate it to the same character '=' as before.

If you do not want any URLs posted, consider translating the "http://" and "www." portions of a URL to the '=' character.

On the site I am administering, we only want to encourage discussions on the topics and not have users pointing to other websites, so this step is effective for our purposes. If your website does depend upon sharing URLs, be careful using the Word Censor translation for this purpose.

If you look at your spam postings, you may notice some patterns in the inappropriate URLS containing portions like "www.kxmodels.com" or "fun.model.com". You can enter the word censor filters "*kxmodels.*" or "*.model.*" to eliminate the significant portion of these types of URLs.

4. For each Spam posting, take a look at the poster's email address. You may notice that there is a pattern of spam email accounts. I had received many spam postings from users with email accounts from server addresses ending in certain East European countries.

In order to discourage these groups of users, I used the 'Ban Control' feature on the User Admin set of tools and blocked all email addresses with these extensions using patterns such as:

*@*.ru
*@*.cz
*@*.tw
*@*.cr

I also received a few from very specific website email accounts. If I noticed a pattern in the postings, I added these to the banned lists.

The users that post spam messages typically create an account, get the verification from their email, and then never visit the site again with the same account. So, this approach will discourage them from taking the time to set up a new fake email on a different mail server everytime that they want to post to the site.

4. I have not blocked any IP address at this time. I have noticed that some spam posters are using an IP address from a secure server that cannot be traced. Or the path of servers used is so convoluted, they could switch to another server if they chose to in the future. Fortunately, my site is not being jammed by a particular IP address at this time, so I do not have to worry about it.

5. The next part of the strategy is to let the spammers know that they are wasting time posting on your website.

The word censor automatic word translation provides immediate feedback that their attempt to post URLs and increase their Search Enginee metrics are not effective on your site.

For each of the spam notices (if they are few), I replace the content with a note indicating that the message posted was inappropriate. The user account is deactivated only if they re-post more inappropriate information.

Only a few of the spam notices will be kept on the forums with this message. All of the others are deleted.

6. The last part of the strategy is to reactivate the forums with a message to the users indicating that changes have occurred. Start monitoring the activity and update your word censor list, email pattern blocking, and IP address blocking if needed.

I would like to automate this approach with the phpBB services and have the toolset automatically delete a posting if the Word Censor list has detected a number of words or phrases that exceed a specified threshold.

Although the website I am administering has been using this strategy for just 2 months, there has been a significant reduction in the amount of spam postings.

The strategy discourages the behavior and hopefully will have the spam groups take my website URL off of their list of sites to visit in the future!

When the website looks like the spammers have let us alone, I may reactivate some of the potential user email address patterns. But, with caution.

I hope these ideas help you manage your websites and develop a strategy that is simple and effective. If you have any ideas for enhancing this approach, let us know!!!

Thanks :)

Last edited by Zenkakuji (2007-12-25 8:28 pm)

fspamlist · 2008-01-08 2:21 am

This is a quality post. Well written and is very informative to newcomers to the forum spam battle.

Thank you for sharing.

TeMerc · 2008-01-08 6:28 am

Welcome to the forums Zenkakuji.

I can tell you what I did for my site. It was simple really.

I was getting upwards of a dozen or so spambot regs per day. Very annoying.

I've always had admin approval for registration too. I'd seen so many other phpBB admins\mods dealing with them registering and dropping links\comments I decided this was best for me. But still they came and registered and it made me mad, real mad.

After this going on for months and months I decided, upon realizing that well over 75-80% of them were using web based free email domains to block them all. Every one I can find. I also began blocking all sorts of TLDs as well. .ru, .cn, .hk and many others.

My spam regs dropped by probably 95-98% almost immediately. I get maybe one or two inquiries per month from users who can't register with YaHoo!, Hotmail or what ever. Sometimes if I get a mention in a newsletter its a few more. But once I email them back with an explanation, they totally get it and join using their ISP based email.

If someone decides not to join, so be it. It's saved me so much time I can't begin to tell you.

Yeah, its a bit extreme, but it works fine for me.

I do however plan on loosening the restrictions some once I upgrade to phpBB 3x, which will be soon. We'll see how that works out.

Of course now with sites like this and fspamlist, I almost want to get more spammers just to add to the database.

Which is something Ill be able to do here because the new phpBB logs reg IPs.

PuenteVista · 2008-01-08 10:54 pm

If you restrict yahoo gmail hotmail signups, you might mention in the registration agreement that they can email you directly with that name and have their account signed up personally.

It's not that big of a hassle to do this for new registrants unless you have a huge forum.

Also, anytime you use an email in a public format, make it in a jpg or gif format to prevent spam harvesters.

fspamlist · 2008-01-09 3:25 am

PuenteVista wrote:

If you restrict yahoo gmail hotmail signups, you might mention in the registration agreement that they can email you directly with that name and have their account signed up personally.
It's not that big of a hassle to do this for new registrants unless you have a huge forum.
Also, anytime you use an email in a public format, make it in a jpg or gif format to prevent spam harvesters.

This is also a good point..

More good reading here about blocking spammers: http://www.phpbbhacks.com/forums/stoppi … 67553.html

Something that I got that most out of was the reverse CAPTCHA colors modification for phpbb2. It inverts the black and white colors of the captcha to make it harder for the spammers to decipher.. When I used it.. it DRAMATICALLY reduced spammers.

Read about it here: http://www.phpbbhacks.com/download/6667

Last edited by fspamlist (2008-01-09 3:26 am)

bookwise · 2008-01-14 8:12 pm

Zenkakuji wrote:

...
The "phpBB" forum toolset contains a "Word Censor" feature. This allows the administrator to enter words that are automatically translated when posted & viewed on the website.
...
3. Many spammers are posting URLs to increase the posting metrics for search engines. If you notice that you are receiving a lot of unwanted website URL postings, consider adding word translations for portions of the URLs.
Example: to eliminate the viewing of a website from a Russian server,
enter '*.ru*' into the Word Censor tool and translate it to the same character '=' as before.
If you do not want any URLs posted, consider translating the "http://" and "www." portions of a URL to the '=' character.

Please give more details on how you are able to use the word censor to disable the urls.

I have put many variations of the parts of url code into the censor but the censor is not making the substitutions.

My primary concern is with the pornographic images. I have tried putting IMG and variations with wildcards into the censor but nothing is working.

I think that one of the problems is that the code seems to be all in a bunch with no spaces or line breaks. I concluded it was formatted that way to defeat the word censor.

I have made extensive use of the censor on the text of the spam posts. Please see my other post today for details.

Thanks for your great post. I will implement many of your suggestions today.

Have a blessed day!

Zenkakuji · 2008-01-15 1:42 am

bookwise wrote:

Please give more details on how you are able to use the word censor to disable the urls.

I would like to thank everyone for sharing additional ideas for enhancing the strategy that I am currently using.

Regarding the request from bookwise on how to use the Word Censor feature to disable URLs, this is how I accomplished it:

1. In the Administrative tools of phpbb, I disabled the HTML tags feature and Bulletin Board codes feature.

The HTML option is listed in "General Adminstration" under the "Configuration" menu item. In the section "User and Forum Basic Settings", there is a yes/no option to allow HTML tags. Click on the "no" option! While you are in this section, you may want to click on "no" for the "BBCode" option as well.

The HTML option disables the ability to use HTML for posting links.

The BBCode option disables the Bulletion Board code language that includes the "URL" keyword feature as well as the others.

2. Using the "General Admin" / "Word Censors" tool, patterns can be added for the BBCode as desired.

Examples:

Word Replacement
*url* =
*.co.* =
*.wmv* =
*.xhost* =
*nude a* =
*www* =
*www.* =
*xxx* =
http://* =
adult 18+ =

I have included some word translations from my list that I consider inappropriate for my site, but still are not offensive for this forum (i hope!).

The word censoring feature is nice because it allows for pattern matching using the asterisk character as a prefix, suffix, or both. Also, it is possible to match a multi-word phrase with spaces in between each word.

The replacement character used on my site is always the same single character. This is done so the search engine bots looking at my pages are not able to see any of the links posted. Thus, the effort in posting an inappropriate post on my site does not result in any benefit to getting these links on a search engine list.

After adopting the strategy, there were initially still many inappropriate posts until we enhanced the email address blocking. After this, we noticed that the spam post content started changing. The posts were shorter and contained less and less inappropriate content. It seemed that the spammers were testing out how effective our method was.

Every time I notice an inappropriate post, I evaluate the new content to enhance the word censor list as well as the email address blocking patterns.

At this point, we are noticing only 1 spam post every 4 days. This is an amazing reduction, but my goal is ZERO!!! Is this a fool's dream? I hope not!

We are discussing changing our account policy to allow only invited guests to join. This will reduce our growth rate of members, but allow for quality content to be read by all of the forum members.

For the next few months, I am going to keep track of our spam posts and maintain some metrics. If the trend is an increase, then we will decide on this membership policy change.

Thanks again to everyone for providing additional ideas to this topic. I greatly appreciate your help and advice.

Last edited by Zenkakuji (2008-01-15 1:47 am)

Captain Red · 2008-01-16 10:40 am

Might tweaking the robots.txt file so that google(for one) didn't index the forum and/or userlist cut down on some of the spam?(I'm looking into doing that now)

Zenkakuji · 2008-01-16 5:47 pm

Captain Red wrote:

Might tweaking the robots.txt file so that google(for one) didn't index the forum and/or userlist cut down on some of the spam?(I'm looking into doing that now)

Thanks for the suggestion about using the robots.txt file. I was not aware of this capability so I googled the topic and found a simple description and examples at:

http://www.robotstxt.org/orig.html

I will be using this approach as well, but I am not sure if the spammers would be aware that a particular site has these options set. So, I think the visual feedback from the Word Censor translation is beneficial to have in place as well.

Thanks for the tip!

Captain Red · 2008-01-17 2:00 am

Zenkakuji wrote:

Captain Red wrote:
Might tweaking the robots.txt file so that google(for one) didn't index the forum and/or userlist cut down on some of the spam?(I'm looking into doing that now)
Thanks for the suggestion about using the robots.txt file. I was not aware of this capability so I googled the topic and found a simple description and examples at:
http://www.robotstxt.org/orig.html
I will be using this approach as well, but I am not sure if the spammers would be aware that a particular site has these options set. So, I think the visual feedback from the Word Censor translation is beneficial to have in place as well.
Thanks for the tip!

You're welcome! And thanks for the link.:¬)

I think a lot of the userlist spam, at least, is predicated upon the idea that the links they shove in there will be indexed by Google and the like and thus raise the spammer's page rank. My theory is that the spammers trying to do this will build in a check to make sure that what they post will get indexed, and not just collect dust on some profile page that no human is likely to ever see.

It's not hard to check... try adding "robots.txt" to the end of nearly any url you surf to.(I'm trying to convince you... I'm trying to convince myself more)

I can't argue with a layered approach, though... which reminds me of a very long post and its associated thread a phpbb's site. It goes into strategies for fighting spambots and spamhumans, mostly directed around modifications you can add to phpbb2. Still, the ideas behind those mods might be valuable for non-phpbb users as well.

Captain Red · 2008-01-17 4:44 am

Another thought struck me: nofollow links. It ties in with the robots.txt idea, but you don't have to know the names of all the crawlers you'd like to block, and it involves more mucking about in the backend.

Adding rel="nofollow" to the html in links will cause google msn, and yahoo not to follow those links(and thus not raise the rankings of the linked sites). The problem is altering the code behind the BBCode to translate links to include the rel="nofollow" bit.

There's a thread on doing this in phpbb, and it includes the code to make the change.

pavemen · 2008-01-17 5:29 pm

I have a forum with over 13,000 registered users and nearly 1 million posts. Its based on an old phpBB 1.2.1 version. I've modded it so much that its got many of the newer features of phpBB2 and vBulletin.

I've begun using this site's spam list of disallowing registrations when matching IP, email or usernames are found. Its working great so far. I add data to the list when new info is encountered. This is a manual process after the spammer has posted,this guarantees that the addition is legit.

I used to disallow the big free email accounts and asked users when this happened to use their ISP email. I did allow users to change back to a free account after registration was complete.

I also modded the registration code to store the original email (even after the user changed it in their profile) as well as the IP and hostname used at the time of registration. It makes it much easier to deal with abuse reports when you have this info at the time of registration. Along with the timestamp, even dynamic IPs can be back traced to a specific users account when filing abuse claims.

I use a basic robots.txt file, but have not updated it in years. I would rather have my site show up higher on search lists and deal with the slight increase in spammers. Traffic is traffic when it comes to selling advertising.

Also, once a spammer has posted, I use the phpBB word censor (remember, this is the one based on v1.2.1) to add the users domain, e.g. topsexsearch, and replace it with "spammer". That way all future links and images do not work.

I do not even use CAPTCHA and spam has dropped to 2-3 a month now.

i did setup my registration code to email when checking against this site. Some days, I'll get 20-25 emails showing that this list here stopped registrations. I think that its more than likely the same person/bot making multiple attempts, but it is working.

I have also made it very easy for users to help report spam. They are also very willing to help curb it, so they report it regularly. To avoid multiple reports, I now flag the post as reported, to reduce the chance of people reporting it over and over.

Zenkakuji · 2008-04-12 2:28 pm

I would like to share our recent addition to minimizing spam on our websites that is proving effective.

During the registration process, we already ask the user to re-enter the graphically displayed code. This helped minimize spam entrants, but it seems like there are tools out there that allow the spambots to detect and enter the valid codes.

I got this idea when I was watching a scene in The Davinci Code movie recently in which the Robert Langdon was asked 3 questions to be allowed entry into the French villa.

With this in mind, I added two simple fill in the blank phrases that are relevant to the focus of the websites. If the user enters the right word, they are allowed to request an account. If not, they are just sent back to the main page.

The fill-in-the-blanks should be obvious to the inidividuals interested in joining your forum. For spammers who are not in this country, or who use English as a second language, or are using spambots to access your site, then this additional set of guards will discourage their attempts.

For example,

Motorcycle is a hot ___. (rod) for a website about motorcycles
Red ___. (Sox) for a website about baseball or sports
Paula ___. (Abdul) for a website about American Idol
John ____. (Deere) for a website about farm equipment.

Choosing specific words that would be obvious to those interested in your website but challenging to random visitors would help minimize unwanted guests. Make sure you choose things that are obvious.

We explain that we are asking these questions to help eliminate unwanted spam posts so it is clear to new visitors why they are being quizzed.

After trying this approach on two websites, we have had unbelievable success with no spam. We are still incorporating the ideas offered in the above approach and blocking specific IP addresses detected from before. But, the random spam visitors we were getting has been eliminated (for now!).

Please consider this approach. It would be nice if this feature was included in a future PHP release for everyone to utilize.

Hope your efforts are going well....

the123king · 2008-04-12 3:10 pm

Irf you use PhpBB versions 2.0.x i would STRONGLY reccomend downloading the Anti-Spam ACP. I will admit that the version i have posted was writtem by me and has StopForumSpam.com intergration in it, but it is very effective and contains many different options for getting rid of spammers. I would strongly reccomend looking down the downloads page here too. Ir contains mant MODs to prevent bots registering.

Anyway, here the link to the Anti-Spam ACP V2.2.0. Version 2.0.9 (which is the version before StopForumSpam intergration) can be downloaded from Lithium Studios

Last edited by the123king (2008-04-12 3:10 pm)

#1 2007-12-25 8:18 pm

A Strategy for Discouraging SPAM

#2 2008-01-08 2:21 am

Re: A Strategy for Discouraging SPAM

#3 2008-01-08 6:28 am

Re: A Strategy for Discouraging SPAM

#4 2008-01-08 10:54 pm

Re: A Strategy for Discouraging SPAM

#5 2008-01-09 3:25 am

Re: A Strategy for Discouraging SPAM

#6 2008-01-14 8:12 pm

Re: A Strategy for Discouraging SPAM

#7 2008-01-15 1:42 am

Re: A Strategy for Discouraging SPAM

#8 2008-01-16 10:40 am

Re: A Strategy for Discouraging SPAM

#9 2008-01-16 5:47 pm

Re: A Strategy for Discouraging SPAM

#10 2008-01-17 2:00 am

Re: A Strategy for Discouraging SPAM

#11 2008-01-17 4:44 am

Re: A Strategy for Discouraging SPAM

#12 2008-01-17 5:29 pm

Re: A Strategy for Discouraging SPAM

#13 2008-04-12 2:28 pm

Re: A Strategy for Discouraging SPAM

#14 2008-04-12 3:10 pm

Re: A Strategy for Discouraging SPAM

Board footer