Adding to the database ?

paul · 2009-11-29 1:10 pm

Should the request to add something to the database really be a POST method ? Can it also be a GET method ?

I use the following in PHP;

file_get_contents('http://www.stopforumspam.com/post.php?username='.$name."&ip_addr=".$ip."&email=".$mail."&api_key=mykey");

which is technically a GET... will this work ?

paul · 2009-11-29 5:07 pm

Apparently it doesn't work if it is a GET... would it be much trouble to "fix" this as it makes it a lot easier to program than a POST method in PHP (not that I'm an expert).

pedigree · 2009-11-29 5:10 pm

from

http://php.net/manual/en/function.file-get-contents.php

<?php
function file_post_contents($url,$headers=false) {
    $url = parse_url($url);

    if (!isset($url['port'])) {
      if ($url['scheme'] == 'http') { $url['port']=80; }
      elseif ($url['scheme'] == 'https') { $url['port']=443; }
    }
    $url['query']=isset($url['query'])?$url['query']:'';

    $url['protocol']=$url['scheme'].'://';
    $eol="\r\n";

    $headers =  "POST ".$url['protocol'].$url['host'].$url['path']." HTTP/1.0".$eol.
                "Host: ".$url['host'].$eol.
                "Referer: ".$url['protocol'].$url['host'].$url['path'].$eol.
                "Content-Type: application/x-www-form-urlencoded".$eol.
                "Content-Length: ".strlen($url['query']).$eol.
                $eol.$url['query'];
    $fp = fsockopen($url['host'], $url['port'], $errno, $errstr, 30);
    if($fp) {
      fputs($fp, $headers);
      $result = '';
      while(!feof($fp)) { $result .= fgets($fp, 128); }
      fclose($fp);
      if (!$headers) {
        //removes headers
        $pattern="/^.*\r\n\r\n/s";
        $result=preg_replace($pattern,'',$result);
      }
      return $result;
    }
}
?>

paul · 2009-11-29 5:50 pm

I know that it is possible (and found it on Google as well), but why make it so difficult since the other API's are all GET based.

Personally I don't see any reason this API would need a POST (it doesn't accept large values, neither will it be used from a form... which are basically the two reasons for choosing POST over GET).

In addition, it is not a big problem in most languages to make it work for both POST/GET methods, so you don't have to break anything.

(For the rest, great service !)

zaphod · 2009-11-29 8:50 pm

There is a very good reason for not using GET/Query on the input to the SFS database.

A skript kiddy, with a set of compromised servers, aka zombies, could in moments, bloat the database to an unmanageable size, with very little programming talent needed.

GET is a dangerous place to get input from, as witnessed by alot of the blocking found at the bottom of the page here (especially thick at the end of the month, very sparse at the beginning as the logs get switched).

So, IMHO, it would be better to stick with the POST method we are using for input to the database.

Zap

MysteryFCM · 2009-11-29 9:47 pm

Zap,
You do realise the same data can be passed extremely simply via POST? (all it takes is a simple change to whatever script they're using, and most provide facilities for both GET and POST, with the attacker only needing to select the respective check box or change a 1 to a 0 or whatever, in a config file.

zaphod · 2009-11-29 9:54 pm

Yes, I do realize this. But so far, with much monitoring, I have not seen them using POST/Header.

Zap

paul · 2009-11-29 10:16 pm

zaphod wrote:

There is a very good reason for not using GET/Query on the input to the SFS database.

Security by obscurity !

zaphod wrote:

A skript kiddy, with a set of compromised servers, aka zombies, could in moments, bloat the database to an unmanageable size, with very little programming talent needed.

The code for doing the same is even posted in this same thread... again security by obscurity (and even a very bad one).

zaphod wrote:

GET is a dangerous place to get input from

GET is no different from POST... and the URL doesn't prove anything. By the way, how do you think they're spamming forums... indeed by using the POST method ! If the API by itself was a target, it would hit you before you would even know it ! (if you want to protect that, I've much better measures in mind).

zaphod wrote:

So, IMHO, it would be better to stick with the POST method we are using for input to the database.

The above code is in place, so I'm contributing at this very moment to SFS, but I still can't find myself in the arguments.

zaphod · 2009-11-29 10:31 pm

paul wrote:

GET is no different from POST... and the URL doesn't prove anything. By the way, how do you think they're spamming forums... indeed by using the POST method ! If the API by itself was a target, it would hit you before you would even know it ! (if you want to protect that, I've much better measures in mind).

Actually, GET is a little different from post, as in alot of hacked systems do not allow modification of headers to affect a POST method. Also oftimes headers have allready been sent by the time the hostile script can hook execution.

All that aside, queries are ofttimes logged in publicly viewable areas along with access logs, as a defacto matter of policy. Meanwhile POST is usually never exposed. Having an API key get exposed due to a misfire would be very detrimental. So in a small way, it's security through obscurity.

Am I totally wrong in this?

Zap

P.S. BTW, Security through Obscurity only works for a little while. For an explanation of a real world example of this, listen to the podcast here http://twit.tv/sn213 . It's actually about the most thought provoking podcast I've ever heard.

paul · 2009-11-29 10:45 pm

zaphod wrote:

Actually, GET is a little different from post, as in alot of hacked systems do not allow modification of headers to affect a POST method.

Give me an example... if a bot is running on my PC, why wouldn't he be able to create whatever headers on a POST request ? (unless you make a false assumption on where the bot is running... see below).

zaphod wrote:

Also oftimes headers have allready been sent by the time the hostile script can hook execution.

So you presume a hostile script is being part of an existing script... and not some independent code that is running on an infected system and can GET/POST whatever it wants ?

zaphod wrote:

Am I totally wrong in this?

Yes... apart from the logging argument. GET requests have their parameters revealed in proxy logs, while this isn't the case for a POST request.

BUT ! The only problem is that the way how the add API on SFS works... is wrong ! The API also requires the parameters on the posting URL, and not like a normal POST request works where the data is provided in the request body (and not the headers) !

So sorry, but you're completely wrong... and even fail in security by obscurity.

pedigree · 2009-11-29 11:03 pm

Looking at the code (over the last months) I think a lot of the POST vs GET comes down to Russ' awesome desire to get this site up and running and using it as his first real live PHP project. I have replaced much of the code in this period, removing the XSS and SQL injections, fixing up mysql to reduce database load etc.

There really isnt any technical issues with why I cant add it to the codebase but Ill need to ask Russ as its his code and he mightve done it for a reason that hes yet to share.

So, lets all calm down a little and see what Russ says. If he has no onjections, then its only a couple of small changes to a couple of files.

zaphod · 2009-11-29 11:10 pm

paul wrote:

zaphod wrote:
Actually, GET is a little different from post, as in alot of hacked systems do not allow modification of headers to affect a POST method.
Give me an example... if a bot is running on my PC, why wouldn't he be able to create whatever headers on a POST request ? (unless you make a false assumption on where the bot is running... see below).

I am not talking about your PC. I am talking about server based bots, and/or server proxy bots. Many of those cannot make custom headers. PHP based botnets are a rising scourge of the internet.

paul wrote:

zaphod wrote:
Also oftimes headers have allready been sent by the time the hostile script can hook execution.
So you presume a hostile script is being part of an existing script... and not some independent code that is running on an infected system and can GET/POST whatever it wants ?

Many scripts are hooked into an exploit at the moment of execution via RFI to keep from leaving traces on the exploitable host. I have records of this. There are even botnets that exist only in RAM and re-execute themselves every execution timeout.

paul wrote:

zaphod wrote:
Am I totally wrong in this?
Yes... apart from the logging argument. GET requests have their parameters revealed in proxy logs, while this isn't the case for a POST request.
BUT ! The only problem is that the way how the add API on SFS works... is wrong ! The API also requires the parameters on the posting URL, and not like a normal POST request works where the data is provided in the request body (and not the headers) !
So sorry, but you're completely wrong... and even fail in security by obscurity.

I do so disagree with you. You should go argue that with Steve Gibson, re:security through obscurity.

Open standards are far more secure, as they are open to peer review. Listen to the netcast mentioned above on the now well cracked GSM standard (all it takes is about $2500 in equipment, and you can listen to any cellphone you like now.) This is a major case of FAIL! in security through obscurity.

So, if it's so easily doable in post, and so insecure, why do you want it to be passed in query again? I am confused.

Zap

pedigree · 2009-11-29 11:21 pm

unless someone really screws up with php and leaves register_globals and magic_quotes_gpc on, RFI is quite difficult with POST's vs GET's

paul · 2009-11-30 7:53 am

paul wrote:

BUT ! The only problem is that the way how the add API on SFS works... is wrong ! The API also requires the parameters on the posting URL, and not like a normal POST request works where the data is provided in the request body (and not the headers) !

I'll correct myself on this... the code is correct, and having the query on the request body of the POST (shame on me, and my apologies).

zaphod wrote:

I am not talking about your PC. I am talking about server based bots, and/or server proxy bots.

I haven't done a statistically correct analysis yet, but the spam bots I've seen hitting my server are all on non-fixed IP-addresses (ie. normal "dial-up" users) that don't even have a webserver running. These are infected PC's that have some real binaries running on them, and definitely not an infected PHP script.

zaphod wrote:

Open standards are far more secure, as they are open to peer review.

Indeed, that's why we're having this discussion (which I initially didn't started regarding a security issue, but pure for POST being a "very little" bit more annoying than a GET.

zaphod wrote:

I do so disagree with you. You should go argue that with Steve Gibson, re:security through obscurity.

The fact is that GET is not more secure as POST... both travel unencrypted over the network. The fact that you don't see it on the URL therefore gives you a false sense of security (which is maybe more correct than calling it security by obscurity).

pedigree wrote:

So, lets all calm down a little and see what Russ says. If he has no onjections, then its only a couple of small changes to a couple of files.

No problem... it was just an easy question on POST vs. GET, and as I stated I have the POST running. It's only that I like to understand certain decisions, and for this I couldn't find a real valid one.

pedigree · 2009-11-30 9:06 am

And I even made all the changes so that you can GET submit via a url now....

zaphod · 2009-11-30 9:18 am

Yes, in my opinion, the argument between me and Paul was purely forensic.

Musing here, I wonder if submitting both via post and get would be more secure in some way if both were done simultaneously. Perhaps one passing an encrypted version of the other.

Just daydreaming.

Zap

paul · 2009-11-30 12:02 pm

zaphod wrote:

Yes, in my opinion, the argument between me and Paul was purely forensic.

Indeed, if we don't talk about it, it will never be questioned/improved.

zaphod wrote:

Musing here, I wonder if submitting both via post and get would be more secure in some way if both were done simultaneously. Perhaps one passing an encrypted version of the other.

If you would encrypt, there's no reason to provide a second unencrypted one (if one can break the encryption, they could also send the unencrypted one), and it will double the load.

If you would really want to make it more secure (without going to SSL), one could add an additional checksum (calculated with a second personal/private key) on the provided data, or just check the remote address of the request (and match this against a preregistered IP-address linked to the API key - but this limits testing capabilities).

Alex Kemp · 2009-12-03 1:07 am

Hi there

I came here to check out details on the POST, having just registered (not yet submitted anything, only requesting, which is working fine).

I was confused, as the URL offered looked like a GET request, yet it said "POST". From the other replies, I now understand that it is a URL with GET parameters that is POSTed. Hmm. (Thanks to pedigree for making the GET possible now).

My main security concern is the ease that the current schema allows for the API key to be picked up by man-in-the-middle snooping; proxy logs has already been mentioned as one possible source. POST would make perfect sense, if:

1 POST parameters were used, rather than query parameters in the URL
2 Specifically, the API key was NOT in the URL
3 the URL was HTTPS

I can imagine that the extra overhead of a secure URL may pose load concerns as the site becomes more popular, but most certainly a plain-text key should never be sent unencrypted. It's a bit like the bank posting the PIN number for your new Credit Card on the *outside* of the envelope - slightly insecure.

It makes sense to me that Russ (the originator of this site, right?) wanted to keep things simple for users - a laudable intention, and I thank him for it! However, the current schema makes spoofing of API keys much too easy. I sincerely hope that his check-routines compare source-IP with original submission IP to help guarantee viability.

If the current arrangement cannot be changed, may I suggest some form of API-key encryption? I do not have hands-on experience to offer, but I would think that some kind of private-public key arrangement could be worked out. However, a true POST on a https url would make much more sense.

HTH

pedigree · 2009-12-03 1:11 am

Ive not uploaded the code that allows the GET yet, Ive been way to busy with paying work.

However, the current schema makes spoofing of API keys much too easy.

Wrapping it in layers of code/platform dependent encryption isnt going to make any real impact as we arent seeing apikey abuse. If any is pointed out, then we can look into something

Alex Kemp · 2009-12-03 2:02 am

pedigree wrote:

we arent seeing apikey abuse

Yes, fully understand resisting extra coding that may never be necessary - been there, done that. This one, however, is glaringly obvious. Such an obvious way to compromise the value of this site:- pollute it with hundreds/thousands of false values before anyone can notice. Value built up over years can be destroyed in seconds due to lack of action.

I see from another post that the parameters can be true POST values (body of content) and do not need to be in the URL. That answers half of my concern. If an alternative HTTPS url could be made available, that (for me) would be a reasonable solution, with no need for custom encryption.

Of course, programming the cURL code to make the https call would be a nightmare, but that is my problem, not yours.

PS
I'm trying to promote stopforumspam.com, and want as many other forums to add to the DB as possible. Code available via a php Class. phpBB2 Mod included. Currently only for GET. POST will be added later:
http://download.modem-help.co.uk/non-mo … DNSBL-RBL/

pedigree · 2009-12-03 10:16 am

cURL isnt a huge issue, PHP handles that quite well. Its .NET (m$ api hash functions are still borken), java, ruby and people that dont have curl

paul · 2009-12-03 6:43 pm

If the API key would ever become an issue, I think the most easy solution is to provide the IP-address of your hosting along with the API registration so that the add request can check if it comes from the correct source.

For a man in the middle attack, POST is no more secure than a GET... with the only exception of proxy logs (but again, if anyone adds spammers from his hosting, proxies are normally not used).

BTW, the POST is not providing the key on the URL... I was also first mistaken by that, but if you look carefully at the code, you'll notice that the query part of the URL provided to the function is tranferred to the POST body.

Alex Kemp · 2009-12-03 7:23 pm

Hi paul.

paul wrote:

the POST is not providing the key on the URL

It's the presence of the `?' in the example on the `API' page. Usually no question-mark in a POST request, only in the URL of a GET. We know that there are two of us who got the wrong end of the stick, there may be many more.

I'm much more settled now that I know that the submission can be a POST with request-body parameters. IMO no GET should be offered; far too likely to lead to false submissions on stolen API keys. The suggestion of an alternative secure URL is simply an obvious extra step. Little extra PHP coding for SFS, mostly web setup:

https://www.stopforumspam.com/post.php

Possible?

pedigree · 2009-12-03 8:55 pm

https is pretty much out of the question at the moment, it costs money for a signed cert

Russ · 2009-12-04 1:10 am

Possible if I could turn up apache-ssl somewhere and as indicated a valid signed certificate costs money too.

#1 2009-11-29 1:10 pm

Adding to the database ?

#2 2009-11-29 5:07 pm

Re: Adding to the database ?

#3 2009-11-29 5:10 pm

Re: Adding to the database ?

#4 2009-11-29 5:50 pm

Re: Adding to the database ?

#5 2009-11-29 8:50 pm

Re: Adding to the database ?

#6 2009-11-29 9:47 pm

Re: Adding to the database ?

#7 2009-11-29 9:54 pm

Re: Adding to the database ?

#8 2009-11-29 10:16 pm

Re: Adding to the database ?

#9 2009-11-29 10:31 pm

Re: Adding to the database ?

#10 2009-11-29 10:45 pm

Re: Adding to the database ?

#11 2009-11-29 11:03 pm

Re: Adding to the database ?

#12 2009-11-29 11:10 pm

Re: Adding to the database ?

#13 2009-11-29 11:21 pm

Re: Adding to the database ?

#14 2009-11-30 7:53 am

Re: Adding to the database ?

#15 2009-11-30 9:06 am

Re: Adding to the database ?

#16 2009-11-30 9:18 am

Re: Adding to the database ?

#17 2009-11-30 12:02 pm

Re: Adding to the database ?

#18 2009-12-03 1:07 am

Re: Adding to the database ?

#19 2009-12-03 1:11 am

Re: Adding to the database ?

#20 2009-12-03 2:02 am

Re: Adding to the database ?

#21 2009-12-03 10:16 am

Re: Adding to the database ?

#22 2009-12-03 6:43 pm

Re: Adding to the database ?

#23 2009-12-03 7:23 pm

Re: Adding to the database ?

#24 2009-12-03 8:55 pm

Re: Adding to the database ?

#25 2009-12-04 1:10 am

Re: Adding to the database ?

Board footer