A Thanks to Stop Forum Spam!!

Night Hawk · 2012-02-22 3:51 am

Here I have been finding if you block the one IP for a spammer run a check on the email address(s) used as well. That will reveal often a ton of other spammer IPs!

In just the last two months I've been manually running all the checks to block thousands with the help of the email information for each new spammer that shows up! A great asset for getting rid of "their spamming buddies" as well as the spammer!

Sure enough when going to check up on the next spammer's email address some of those same IPs come up over and over again. When checking on the IPs pooled email addresses being shared between the more organized spammers seems to be the case.

insektenfang · 2012-02-22 12:27 pm

It's the same "people" over and over again, it's just that new email addresses/accounts are generated as required. One IP address may have hundreds or thousands of email addresses associated to it.

bastones · 2012-02-22 2:47 pm

We'd also like to thank StopForumSpam. Our company forums constantly had spammers registering on our forum and since we have installed the StopForumSpam plugin, spammer registrations have greatly been reduced .

Last edited by bastones (2012-02-22 2:49 pm)

kyrian · 2012-02-22 4:55 pm

Thanks a million for your great job and the API.
The component you made for joomla is a "must_have" for admin. No more time lost checking the new registrations on my sites.

Night Hawk · 2012-02-23 3:28 am

insektenfang wrote:

It's the same "people" over and over again, it's just that new email addresses/accounts are generated as required. One IP address may have hundreds or thousands of email addresses associated to it.

With the help this site provides I can confirm some are the same who continue to hang around and watch like keeping tabs as well as network with others! One IP may see hundreds of temp email addresses while one email address sees hundreds of IPs!

Not all the IPs are for the exact same spammers when the IPs are from various locations around the globe! This is why I have to mention they tend to be organized in groups that share email addresses to get new spammers started off. Some of the "work at home" junk mail you may end deleting from your inbox may just be one recruiting method!

handruin · 2012-02-24 3:48 pm

I'd also like to say thank you and share my appreciation for this tool and service. I've been using this for some time now and it has helped my forum considerably in reducing spam accounts. I've made a donation last year and plan to make another one this year. I can only encourage others to send in a donation, even if it's just $1 to show your appreciation to SFS. Every bit can go a long way to keep the site running for all of us who benefit from this.

pedigree · 2012-02-24 4:28 pm

i would love if it every user donated 25 cents per month but donations hardly ever reach triple digits per month. The server costs 99 euro to give you an idea.

I guess lots of people don't think that this site, the service and what we do, as worth a quarter a month...

handruin · 2012-02-25 2:22 am

pedigree wrote:

i would love if it every user donated 25 cents per month but donations hardly ever reach triple digits per month. The server costs 99 euro to give you an idea.
I guess lots of people don't think that this site, the service and what we do, as worth a quarter a month...

Maybe or maybe not. I realize it is worth way more than 25 cents/month. Has there been any good effort to advocate for donations to help keep the site alive and without a membership fee? Is there any kind of reasonable guideline to show folks that even 25 cents goes a long way (or $3/year if you're doing 25 cents/month). I would hate to see this site get into a panic mode of desperation for finances when it's too late.

Is there any kind of model that could offer a paid membership that gains additional features? What about sharing the load in some kind of distributed model to help reduce bandwidth costs? Kind of like how downloads are mirrored for large open source projects.

handruin · 2012-02-25 2:31 am

To practice what I preach, I just sent you guys a donation for $50 USD. Come on folks, show your love for this site and service!

Katana · 2012-02-25 5:01 am

handruin wrote:

What about sharing the load in some kind of distributed model to help reduce bandwidth costs? Kind of like how downloads are mirrored for large open source projects.

Believe me, we'd love to get that working - upstream issues with redis and (I think, may be wrong about this one) facebook's hiphop compiler, however, are preventing it.

John Darkhorse · 2012-02-25 5:47 am

Katana wrote:

handruin wrote:
What about sharing the load in some kind of distributed model to help reduce bandwidth costs? Kind of like how downloads are mirrored for large open source projects.
Believe me, we'd love to get that working - upstream issues with redis and (I think, may be wrong about this one) facebook's hiphop compiler, however, are preventing it.

A "round robin" type system can't be instituted, so that anyone accessing the API gets forwarded to an API mirror chosen at random?

Similar to how the Ubuntu or Sourceforge downloads page sends folks to various mirrors when they click the "download" link.

The "facebook hiphop compiler" shouldn't be involved, since any mirrors would be (theoretically) updating their file(s) regularly, after they're compiled.

Katana · 2012-02-25 3:18 pm

Yes, the hiphop compiler is quite necessary - with it we'd be serving the API using a compiled implementation rather than PHP, which would be loads faster, less memory intensive, and able to handle more concurrent connections. Right now we've hit a blocker with packed strings containing \x00 that's preventing deployment, as it causes the API to puke (without packing the strings, as well, memory use explodes on redis).

The issue though is not with DNS and directing traffic to nodes, but securely mirroring the API to said nodes - there is no way, currently, for us to do this with redis without opening the master up in ways that wouldn't be safe - it would mean that if an end node were compromised, an attacker could dump the master's data and interfere with the API for a significant amount of time; the data wouldn't be permanently lost as it's all stored in mysql as well, but there'd be a significant interruption in service.

pedigree · 2012-02-25 3:35 pm

ssltunnel for a replication path, isnt working. it will work for a day or a week or even a month and will then fail on the client nodes. The only way to get it working again is to restart the server tunnel, which is a pain to do at 3am. Im putting in a layer3 transparent tunnel at the moment, which will (should) allow Redis to replication and thus, allow for dns to provide 4 geographically separate locations for api lookups.

Currently, the api is running from php with a HpHop version running in the background doing tests on newer code. its this way at the moment because I cant run two copies of hiphop on the same machine or chance and issue of a single copy providing two pages. Ill sometimes swap them over and for the better part of the move, no one notices any difference apart from the system loading.

Katana · 2012-02-25 4:07 pm

pedigree wrote:

ssltunnel for a replication path, isnt working. it will work for a day or a week or even a month and will then fail on the client nodes. The only way to get it working again is to restart the server tunnel, which is a pain to do at 3am. Im putting in a layer3 transparent tunnel at the moment, which will (should) allow Redis to replication and thus, allow for dns to provide 4 geographically separate locations for api lookups.
Currently, the api is running from php with a HpHop version running in the background doing tests on newer code. its this way at the moment because I cant run two copies of hiphop on the same machine or chance and issue of a single copy providing two pages. Ill sometimes swap them over and for the better part of the move, no one notices any difference apart from the system loading.

oh, did the blocker with sprintf get taken care of then?

pedigree · 2012-02-25 4:43 pm

From memory, yes. Ill need to revisit that and add a test to the units

John Darkhorse · 2012-02-25 5:04 pm

So, I'm new to all this end of things (but have used your service for years) so bear with me

I take it the file can't be offered as a csv or something?

The master csv file can be hashed with sha1 or something that hasn't been cracked (like md5) and anyone downloading from a mirror could check the hash against the master sha1 sum located here (if they didn't trust the mirror's posted sha1 sum)

Mirrors could update regularly during the day, so everyone'd be within 12 (or less) hours of the master file.

Rsync could be used, so as not to do a complete data pull each time (saving resources on this end).

Just throwing some ideas out there (I have no idea how to implement a secure network of HipHop)

Katana · 2012-02-25 5:19 pm

John Darkhorse wrote:

So, I'm new to all this end of things (but have used your service for years) so bear with me
I take it the file can't be offered as a csv or something?
The master csv file can be hashed with sha1 or something that hasn't been cracked (like md5) and anyone downloading from a mirror could check the hash against the master sha1 sum located here (if they didn't trust the mirror's posted sha1 sum)
Mirrors could update regularly during the day, so everyone'd be within 12 (or less) hours of the master file.
Rsync could be used, so as not to do a complete data pull each time (saving resources on this end).

Just throwing some ideas out there (I have no idea how to implement a secure network of HipHop)

Wouldn't be accurate enough - we get quite a bit of data every few minutes; spammers like to change things up and shift between IPs quite often, and when you consider that IPv4 covers 2^32 (that's 2 to the thirty-second power) addresses (and when IPv6 hits a larger scale, that'll expand it to 2^128 - 2 to the one-hundred-twenty-eighth power)...the scale of the area that a spammer can occupy becomes immense.
Rsync is made for modest synchronization like log files, codebases, etc. It never was designed with database replication in mind, and it wouldn't be able to handle that anyways - that's a job for other tools.
Lookups against a CSV, also, have horrible performance - you're basically seeking through the entire file to see if an entry exists, which in the cases it doesn't - that means you're seeking through the entire file. With as many requests as SFS gets now, even if you divided that by four (assume four api nodes, each one getting an equal portion of traffic) you're still talking about hundreds and hundreds of requests stacking up on top of one another - and if it's a CSV lookup, it's going to overwhelm the node very, very quickly.
This opens up the end-nodes to denial-of-service attacks by spammers themselves simply by making an overwhelming number of API lookups to each. So, that's not really an option.
(plus, that methodology is open to a Man in the Middle attack - checksums mean nothing, you need signed or encrypted transmission methods to prevent MITM)

pedigree · 2012-02-25 8:03 pm

If I was to offer a rsync service of the data, it would be a continual stream of traffic

handruin · 2012-02-27 9:24 pm

What about a model that makes use of something like Cassandra or one of those other NoSQL-based distributed components? Changes at your source could atomically be distributed to other nodes to help scale in a more-linear fashion as you grow. I realize this wouldn't be easy to do overnight, or even possibly in the next year, but maybe it might address an IO contention issue that may be an on-going battle.

Does SFS have to be the center hub for all authentication for reads off the API? Can there be future anti-spam programs (not software) that other "mirrors" might be able to apply for to help the cause?

Katana · 2012-02-27 10:46 pm

Problem with most NoSQL storage is that it is very volatile, and we're definitely depended upon by a lot of sites.
I do hope that at some point though we can get something where we could easily get mirrors set up, but it has to be in a way that doesn't leave us all open to problems where data is lost or can be wiped out.

Do know though that we want this - hell, I'm sure there's hosts that would love to host a local SFS mirror that offers their users unlimited queries or some such. Not saying that'd be guaranteed at all, but that would be a great trade for getting the database set up for HA.

pedigree · 2012-02-27 10:54 pm

MySQL replication works.... It just works, without issue, continually, regardless of what I do to any of the servers.

Redis replication is awful and just stops, even if Im doing nothing. It is the only thing that I dislike about Redis.

handruin · 2012-02-28 4:06 am

Katana wrote:

Problem with most NoSQL storage is that it is very volatile, and we're definitely depended upon by a lot of sites.
I do hope that at some point though we can get something where we could easily get mirrors set up, but it has to be in a way that doesn't leave us all open to problems where data is lost or can be wiped out.
Do know though that we want this - hell, I'm sure there's hosts that would love to host a local SFS mirror that offers their users unlimited queries or some such. Not saying that'd be guaranteed at all, but that would be a great trade for getting the database set up for HA.

What would you estimate is the rough percentage of reads/writes from users (reads = API data requests, writes = API data submits)? If we (users) are more read-heavy, could there be simple read-only mirrors to help offset at least some of the burden? Any and all writes/submits would still come directly to the source (here). Also, how soon is the new data that we submit become available for spam detection?

pedigree · 2012-02-28 12:30 pm

The API is 50:1 read heavy but thats completely out of RAM caches

Katana · 2012-02-28 1:36 pm

handruin wrote:

Katana wrote:
Problem with most NoSQL storage is that it is very volatile, and we're definitely depended upon by a lot of sites.
I do hope that at some point though we can get something where we could easily get mirrors set up, but it has to be in a way that doesn't leave us all open to problems where data is lost or can be wiped out.
Do know though that we want this - hell, I'm sure there's hosts that would love to host a local SFS mirror that offers their users unlimited queries or some such. Not saying that'd be guaranteed at all, but that would be a great trade for getting the database set up for HA.
What would you estimate is the rough percentage of reads/writes from users (reads = API data requests, writes = API data submits)? If we (users) are more read-heavy, could there be simple read-only mirrors to help offset at least some of the burden? Any and all writes/submits would still come directly to the source (here). Also, how soon is the new data that we submit become available for spam detection?

Last question's answer: damn near immediate.
Maybe a minute tops that it takes to be completely available.

pedigree · 2012-02-28 2:07 pm

The second that its in the database, its available for api testing on this site. On remote sides, if Redis replication, is working, it would 1-2 seconds before its available.

#26 2012-02-22 3:51 am

Re: A Thanks to Stop Forum Spam!!

#27 2012-02-22 12:27 pm

Re: A Thanks to Stop Forum Spam!!

#28 2012-02-22 2:47 pm

Re: A Thanks to Stop Forum Spam!!

#29 2012-02-22 4:55 pm

Re: A Thanks to Stop Forum Spam!!

#30 2012-02-23 3:28 am

Re: A Thanks to Stop Forum Spam!!

#31 2012-02-24 3:48 pm

Re: A Thanks to Stop Forum Spam!!

#32 2012-02-24 4:28 pm

Re: A Thanks to Stop Forum Spam!!

#33 2012-02-25 2:22 am

Re: A Thanks to Stop Forum Spam!!

#34 2012-02-25 2:31 am

Re: A Thanks to Stop Forum Spam!!

#35 2012-02-25 5:01 am

Re: A Thanks to Stop Forum Spam!!

#36 2012-02-25 5:47 am

Re: A Thanks to Stop Forum Spam!!

#37 2012-02-25 3:18 pm

Re: A Thanks to Stop Forum Spam!!

#38 2012-02-25 3:35 pm

Re: A Thanks to Stop Forum Spam!!

#39 2012-02-25 4:07 pm

Re: A Thanks to Stop Forum Spam!!

#40 2012-02-25 4:43 pm

Re: A Thanks to Stop Forum Spam!!

#41 2012-02-25 5:04 pm

Re: A Thanks to Stop Forum Spam!!

#42 2012-02-25 5:19 pm

Re: A Thanks to Stop Forum Spam!!

#43 2012-02-25 8:03 pm

Re: A Thanks to Stop Forum Spam!!

#44 2012-02-27 9:24 pm

Re: A Thanks to Stop Forum Spam!!

#45 2012-02-27 10:46 pm

Re: A Thanks to Stop Forum Spam!!

#46 2012-02-27 10:54 pm

Re: A Thanks to Stop Forum Spam!!

#47 2012-02-28 4:06 am

Re: A Thanks to Stop Forum Spam!!

#48 2012-02-28 12:30 pm

Re: A Thanks to Stop Forum Spam!!

#49 2012-02-28 1:36 pm

Re: A Thanks to Stop Forum Spam!!

#50 2012-02-28 2:07 pm

Re: A Thanks to Stop Forum Spam!!

Board footer