API Rate Limiting

Russ · 2009-03-28 1:34 am

We will soon be implementing a rate limiting scheme into the API for checking IPs/usernames/email addresses. I hate to but after analyzing the hits there is a handful of hosts who are hammering it constantly, and I want to make sure the server resources are not being hogged because of it.

The limit will probably be around 1000 API queries per day, which is going to be plenty for most everyone. If you need more than that, let me know and something can probably be worked out.

If your script is integrated with the API, and your host exceeds the daily limit, the server will return a 403 HTTP status code and the output will look like this.

<response success="false">
        <error>rate limit exceeded</error>
</response>

You should be able to code sufficient error handling for this case should it happen.

MysteryFCM · 2009-03-28 1:47 am

I'll code this into the SBST

MysteryFCM · 2009-04-01 7:50 pm

Got this from the temerc.com domain (running the SBST)

"StopForumSpam informed me your daily query limit has been exceeded"

Russ · 2009-04-01 8:27 pm

That's curious because there is no rate limiting happening yet!

bbengineer · 2009-04-01 8:32 pm

There was an issue this afternoon with the SFS server and it was returning 500 errors for a while. Looks like your script interpreted the 500 error as the 403 error being used to respond for over the limit

BBEngineer

MysteryFCM · 2009-04-01 8:34 pm

... except the script is still receiving it now

I'll check the code to ensure it's not a bug in my script

/edit

Got a feeling it's a bug ...... I've got the following code checking the return value, and it's returning absolutely err - nothing;

$bSFSLimit = strpos($fspamcheck, 'rate limit exceeded');
if($bSFSLimit == True ){

... this is acting as though $bSFSLimit is true, even though it's set to false by default.

/edit 2

Nevermind, was checking the wrong bit of code ....... it was caused by the following;

if($bSFSLimit=true){

Last edited by MysteryFCM (2009-04-01 8:49 pm)

pedigree · 2009-04-01 10:00 pm

To give you an idea of numbers, one site was constantly hitting the API over 40,000 times a day

MysteryFCM · 2009-04-01 10:02 pm

Either one exceedingly popular website, or a spambot that really really really wanted to get into it ..... or of course, just someone abusing your API ...

pedigree · 2009-04-01 10:29 pm

They run vbulletin and dont seem to be using my mod as they are hammering us for the same data

Wired · 2009-04-02 4:38 am

= vs ==, always a popular mistake

MysteryFCM · 2009-04-02 4:46 am

hehe definately (normally catch those if I do it, during pre-release testing, but missed it for some reason)

pedigree · 2009-04-02 9:22 am

== vs === still a popular mistake

Russ · 2009-04-12 3:30 pm

Rate limiting is in place now. The allowed number of queries is 5000 per day from a single IP.

kpatz · 2009-04-13 12:49 pm

MysteryFCM wrote:

Either one exceedingly popular website, or a spambot that really really really wanted to get into it ..... or of course, just someone abusing your API ...

I wonder if they're calling the API for every page hit or something.

My site only hits the API on a new registration, and when they activate their account by hitting the link they receive in email. That and when the admin (me) does a lookup. And the lookups are cached locally too. I think we'll stay well under the 5,000/day limit.

Russ · 2009-04-13 2:42 pm

5,000 hits a day is extremely liberal and after looking at the numbers over the past few weeks, no one's hit the limit except for the same 6 or 7 IPs who hammer the API constantly and excessively.

ih8spam · 2009-04-13 3:41 pm

yeah, I would think that even 1000 querys a day would be more than enough.

I'm a member of a board that was getting clobbered by spam until about 2 months ago when they finally put in some anti-spam code .

and I estimated that even THEY were only getting 50-70 spammer sign-ups a day .

Last edited by ih8spam (2009-04-13 3:41 pm)

hhopper · 2009-04-13 5:55 pm

That sounds really excessive to me. I wonder if the ones that are hammering you are spammers themselves trying to jam up your site. I can't imagine any reason for checking your site more than a few times per day. Nobody gets than many attempted logons.

Hop

M-DVD · 2009-04-13 11:12 pm

ih8spam wrote:

yeah, I would think that even 1000 querys a day would be more than enough.
I'm a member of a board that was getting clobbered by spam until about 2 months ago when they finally put in some anti-spam code .
and I estimated that even THEY were only getting 50-70 spammer sign-ups a day.

Russ wrote:

5,000 hits a day is extremely liberal and after looking at the numbers over the past few weeks, no one's hit the limit except for the same 6 or 7 IPs who hammer the API constantly and excessively.

Depends.

If the forum has already implemented the system AntiSpam SFS, then even be very active, never will have ~100 checks by days (unless the site is being hit very hard).

But if a site begining to introduce the system AntiSpam, It isn't normal to check all (or many) users? (a site average may have more than one thousand users).

pavemen · 2009-04-15 6:16 pm

M-DVD wrote:

Depends.
If the forum has already implemented the system AntiSpam SFS, then even be very active, never will have ~100 checks by days (unless the site is being hit very hard).
But if a site begining to introduce the system AntiSpam, It isn't normal to check all (or many) users? (a site average may have more than one thousand users).

As of late, my site will get 150-300 SFS catches per day and I only check IP and email addresses at registration time.

Just short of 16,000 registered members (obviously not all are active, lucky if 500 are regulars) and nearly 1 million posts so its a moderately sized forum.

Of course this does not include the IPs that are blocked at the firewall and never make it to the domain to start with.

M-DVD · 2009-04-16 4:08 am

pavemen wrote:

As of late, my site will get 150-300 SFS catches per day and I only check IP and email addresses at registration time.
Just short of 16,000 registered members (obviously not all are active, lucky if 500 are regulars) and nearly 1 million posts so its a moderately sized forum.
Of course this does not include the IPs that are blocked at the firewall and never make it to the domain to start with.

Yes.

You have:
16000 users total.
500 actives.
15500 inactives

It isn't normal you want to check these 15500 users? (or 10000, or 5000)

--

At first glance, 1000 querys per day seems enough or too much. But not always.

I just wanted to make known this point of view.

Last edited by M-DVD (2009-04-16 4:10 am)

pavemen · 2009-04-16 4:50 am

M-DVD wrote:

pavemen wrote:
As of late, my site will get 150-300 SFS catches per day and I only check IP and email addresses at registration time.
Just short of 16,000 registered members (obviously not all are active, lucky if 500 are regulars) and nearly 1 million posts so its a moderately sized forum.
Of course this does not include the IPs that are blocked at the firewall and never make it to the domain to start with.
Yes.
You have:
16000 users total.
500 actives.
15500 inactives
It isn't normal you want to check these 15500 users? (or 10000, or 5000)
--
At first glance, 1000 querys per day seems enough or too much. But not always.
I just wanted to make known this point of view.

Well, I could check them, but since none of them are spamming, I am not worried about it.

I have methods of dealing with spam, from 14 moderators to easy to use report to mod features and a diligent set of active users, so it is not really a problem. When it does get through, its killed in a few minutes. My word censor is very large with a lot of alternate spellings of the major keywords so sex spam is useless and not legible most of the time.

Other spam is reported quickly and dealt with just as fast.

Even if I wanted to check the other users, 1,000 a day would be fine. I would not want to impact server performance so I could always script the checks to run 200-500 per day and let it run once per day for 2 weeks. That would leave me with enough 'left over' API calls to handle my current registration attempts.

Also, for those few folks that are hitting the API so much that a limit is needed, then for their own server performance, they should download the new SQL database once per day and check against a local list.

zaphod · 2009-04-16 5:38 am

You know, if 10% of 15500 inactive users, were to send their validation codes, within 1 hour...

and each of those robots were to spray 100 messages into your forums...

not only would the load, and fragmentation on your MySQL server be something just this side of epic due to 155,000 new messages in so short a time, but your mods will be picking up the pieces for weeks, if they don't revolt and walk.

My suggestion, if you haven't allready done it, is to ban (or otherwise incapacitate) all the inactives, but do not delete. Each one feels like a bullet that hasn't been put in a gun yet to me.

Zap.
(Glad his forum is barely alive and not popular)

pavemen · 2009-04-18 4:29 pm

The thing is, all of those inactive users have posted at one point in time, be it to the forums or the other portions of the site, so I know they are not spammers.

1-2 times per year I go through the user list and delete any accounts that have not made any contributions and have not logged in in the past 6-9 months

kpatz · 2009-04-18 4:42 pm

When I first implemented my SFS hooks, I only checked new registrations (and still do). Some fishy looking older registrations got checked as well, but I don't see the point in scanning all 15,000 users.

Either they're regular contributers (not spammers), or inactive (not spamming), or spammers (have posted spam/have suspicious registrations). That third category are what gets further scrutiny.

I have my forum set to email me whenever a new user registers (if they pass the SFS checks). My favorite users are the ones who post something right away, that makes my life easy. Check their post(s), immediately I know if they're a spammer or not. It's the ones who register and don't post that take more of my time, since then I start checking IPs, googling usernames, etc.

Sometimes if I see a registration from an unfamiliar country, or a possibly fishy registration that I can't tell for sure is a spammer, I'll flag them as "suspicious" so their posts are queued. 99% of these registrants never seem to return to post for some reason. I bet a lot of spammer registrations are abandoned without ever being used.

skippybosco · 2009-04-21 7:27 am

If you are developing your own SFS checks you should consider locally caching data locally as a part of the lookup process similar to what pedigree does in the VB plugin. It will allow you to query your local data store first before making a round trip to SFS servers. Setting your code to refresh every 'x' hours/days should provide a better experience for you (quicker check times) and for SFS (reduced load to server)

#1 2009-03-28 1:34 am

API Rate Limiting

#2 2009-03-28 1:47 am

Re: API Rate Limiting

#3 2009-04-01 7:50 pm

Re: API Rate Limiting

#4 2009-04-01 8:27 pm

Re: API Rate Limiting

#5 2009-04-01 8:32 pm

Re: API Rate Limiting

#6 2009-04-01 8:34 pm

Re: API Rate Limiting

#7 2009-04-01 10:00 pm

Re: API Rate Limiting

#8 2009-04-01 10:02 pm

Re: API Rate Limiting

#9 2009-04-01 10:29 pm

Re: API Rate Limiting

#10 2009-04-02 4:38 am

Re: API Rate Limiting

#11 2009-04-02 4:46 am

Re: API Rate Limiting

#12 2009-04-02 9:22 am

Re: API Rate Limiting

#13 2009-04-12 3:30 pm

Re: API Rate Limiting

#14 2009-04-13 12:49 pm

Re: API Rate Limiting

#15 2009-04-13 2:42 pm

Re: API Rate Limiting

#16 2009-04-13 3:41 pm

Re: API Rate Limiting

#17 2009-04-13 5:55 pm

Re: API Rate Limiting

#18 2009-04-13 11:12 pm

Re: API Rate Limiting

#19 2009-04-15 6:16 pm

Re: API Rate Limiting

#20 2009-04-16 4:08 am

Re: API Rate Limiting

#21 2009-04-16 4:50 am

Re: API Rate Limiting

#22 2009-04-16 5:38 am

Re: API Rate Limiting

#23 2009-04-18 4:29 pm

Re: API Rate Limiting

#24 2009-04-18 4:42 pm

Re: API Rate Limiting

#25 2009-04-21 7:27 am

Re: API Rate Limiting

Board footer