80legs web spider

ih8spam · 2010-08-31 4:51 pm

These guys seem to be up to no good

my board has had 90 visits in the last 48 hours,
with 62 unique IPs/hostnames, it seems to be reading my Robots.txt file at least some of the time, but it's doing some things that the big search bots (like Google) don't do .

Their page is here: www.80legs.com/spider.html

Here are some highlights from my server logs:

ip68-228-61-252.tc.ph.cox.net - - [29/Aug/2010:21:48:13 -0400] "GET /robots.txt HTTP/1.1" 200 1245 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

ip68-228-61-252.tc.ph.cox.net - - [29/Aug/2010:21:48:14 -0400] "GET / HTTP/1.0" 200 37498 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

71.234.174.18 - - [30/Aug/2010:01:53:27 -0400] "GET /./memberlist.php?mode=group&g=2&sid=03642ce39c2f908d2af7f424096c95aa HTTP/1.0" 200 13336 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

mail.inside-out-consulting.com - - [30/Aug/2010:01:58:49 -0400] "GET /_portal_images/flag/de.png HTTP/1.1" 404 1252 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

75.68.97.29 - - [30/Aug/2010:13:16:22 -0400] "GET /./memberlist.php?mode=viewprofile&u=2&sid=03642ce39c2f908d2af7f424096c95aa HTTP/1.0" 200 13373 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

68.10.127.165 - - [30/Aug/2010:13:51:20 -0400] "GET /./search.php?author_id=57&sr=posts&sid=03642ce39c2f908d2af7f424096c95aa HTTP/1.0" 200 31082 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

75.74.216.184 - - [30/Aug/2010:22:51:18 -0400] "GET /././index.php?sid=2ed55dcf8780903473f7118a8d970cd8 HTTP/1.0" 200 37874 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

I have seen hack-bots add "/././" to directories before, so I think that at least some of their clients are up to no good, so I'm adding their user agent string to my .htaccess file,
I can't see any benefit to allowing them on my site .

Last edited by ih8spam (2010-08-31 4:54 pm)

pedigree · 2010-08-31 4:56 pm

Looks like a perfect candidate for .htaccess blocking to me

ih8spam · 2010-08-31 5:05 pm

Yeah, I have known about them for the past year or so, but they only visited my site 1 or 2 times a month, but they have been hammering my site this week .

Fire God · 2010-08-31 5:19 pm

Ok stupid question how to add to the htaccess file to block these?

My current file blocks only IPs and would love to be able to block this agent and the RIM agent with out blocking legit users.

Thanks in advance.

diabolic.bg · 2010-08-31 5:51 pm

This is a directory traversal attack - ZB Block stopped it without problems. I tried but can not stop it with .htaccess.

ih8spam · 2010-08-31 6:08 pm

Fire God wrote:

Ok stupid question how to add to the htaccess file to block these?
My current file blocks only IPs and would love to be able to block this agent and the RIM agent with out blocking legit users.
Thanks in advance.

Here is a sample of my .htaccess user agent blocking,
you should be able to just paste this into your .htaccess file,
and add more lines, or delete lines as you want .

# block user agents from site access
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (www.80legs.com/spider.html) [OR]
RewriteCond %{HTTP_USER_AGENT} (^Morfeus) [OR]
RewriteCond %{HTTP_USER_AGENT} (^libwww-perl) [OR]
RewriteCond %{HTTP_USER_AGENT} (WebDataCentre.com) [OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/5.0 \(compatible; MJ12bot" [OR]
RewriteCond %{HTTP_REFERER} "onlinedrugstore.doras.us"
RewriteRule ^(.*)$ - [F]

there is a complete tutorial HERE, but they do it slightly different than I do .

ih8spam · 2010-08-31 6:11 pm

diabolic.bg wrote:

This is a directory traversal attack - ZB Block stopped it without problems. I tried but can not stop it with .htaccess.

Yeah, I figured as much. lol
blocking 80legs won't stop ALL directory traversal attacks, but it should stop the ones coming from 80legs.com .

diabolic.bg · 2010-08-31 6:30 pm

Yes, but I put in my htaccess summary rules. I like it. As says in Bulgaria "with only one bullet - two rabbits".

Fire God · 2010-08-31 7:31 pm

Many thanks, I will give it a shot and see what happens.

zaphod · 2010-09-01 12:21 am

Actually, that's a broken traversal attack.

Spider is stoned.

./././ gets you nowhere.
../../../ Recurses.
....//....//....// Makes no bloody sense, but I've seen it too.

Zap

P.S. 80legs in user agent will be persona non grata in beta-sigs 65c for ZBB

ih8spam · 2010-09-03 11:21 am

Here is another search bot that didn't even read "robots.txt", let alone obey it.

it read nearly every page on my site, at one page per second .

Originating IP: 91.205.96.19

User agent string:

Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)

zaphod · 2010-09-03 11:32 am

Meh, wonder if that's related to Purityscan... an old bit of fraudulent malware.

Zap

P.S. puritysearch (as "purity", to catch any purityscan perhaps too) &
purebot now added to banned user-agents in 65c (yet to be released).

zaphod · 2010-09-14 9:52 am

Beta-Sigs 65c has been released.

Zap

Force · 2011-06-03 12:16 am

ih8spam wrote:

Fire God wrote:
Ok stupid question how to add to the htaccess file to block these?
My current file blocks only IPs and would love to be able to block this agent and the RIM agent with out blocking legit users.
Thanks in advance.
Here is a sample of my .htaccess user agent blocking,
you should be able to just paste this into your .htaccess file,
and add more lines, or delete lines as you want .
# block user agents from site access
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (www.80legs.com/spider.html) [OR]
RewriteCond %{HTTP_USER_AGENT} (^Morfeus) [OR]
RewriteCond %{HTTP_USER_AGENT} (^libwww-perl) [OR]
RewriteCond %{HTTP_USER_AGENT} (WebDataCentre.com) [OR]
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/5.0 \(compatible; MJ12bot" [OR]
RewriteCond %{HTTP_REFERER} "onlinedrugstore.doras.us"
RewriteRule ^(.*)$ - [F]
there is a complete tutorial HERE, but they do it slightly different than I do .

Still cant work out where to put this file or where about it should be can you upload one and tell me where in the ftp it should be

nitaonline · 2011-06-03 1:24 am

As with any .htaccess file you put it at the level where you want it to do what you tell it to. If you want it to apply to your entire website, you put it in the webroot (usually named public_html or www or something like that).

jonboat · 2011-06-03 4:57 pm

Better yet... go to Zaphod's site and get ZBBlock.

it catches all kids of crap like this and stops it - even when .htaccess doesn't

Katana · 2011-06-04 2:36 am

nitaonline wrote:

As with any .htaccess file you put it at the level where you want it to do what you tell it to. If you want it to apply to your entire website, you put it in the webroot (usually named public_html or www or something like that).

Use httpd.conf; it's parsed once upon start/reload and stored, unlike ZBBlock or htaccess which are run every page load.

zaphod · 2011-06-04 5:20 am

What if you don't have r00t access on the server? ZB Block was targeted towards those who just have accounts on hosts.

Also, can httpd.conf detect emergent hacking threats? Does it learn where the attacks come from, and engage an automatic defense mode after so many strikes? Does it cache positive IP hits from stop forum spam on registration attempts? Does it work with non-linux servers like IIS, and JanaServer?

No one product is an end-all. A layered approach is far better.

And also you are erroneous in your assumption that ZB Block loads on every page. It can be "hooked" into only the pages that need hacking protection (pretty much anything that accepts input from the URI, or POST), leaving static pages to load without it. Also, unlike .htaccess, it doesn't try to protect resource files, like .gif or .swf that cannot be hacked.

Ya really oughta see what I am up to over there.

Zap

Katana · 2011-06-05 3:03 am

zaphod wrote:

Ya really oughta see what I am up to over there.

Being brutally honest here, the last time I looked at the ZBBlock codebase, I was horrified at the code -- readability of the source is downright zilch for a multitude of reasons, one of them being an absolute lack of indentation within blocks. You'd have a hard time with python which uses indentation to denote blocks, as a method of enforcing good programming practice.

zaphod · 2011-06-05 4:27 am

My first 3 computers would error on indentation. You had to keep track of loops in your head, or use REMarks.

So, that's a matter of coding taste. If you really want to have a fit, try ASM.

Zap

Alex Kemp · 2011-06-05 1:44 pm

Obsidian wrote:

zaphod wrote:
Ya really oughta see what I am up to over there.
...an absolute lack of indentation within blocks

zaphod wrote:

try ASM

Oh gawd, no... where's the playground supervisor?

Last edited by Alex Kemp (2011-06-05 1:47 pm)

Katana · 2011-06-05 4:51 pm

zaphod wrote:

My first 3 computers would error on indentation. You had to keep track of loops in your head, or use REMarks.
So, that's a matter of coding taste. If you really want to have a fit, try ASM.
Zap

No thanks.

The reason I'm mentioning this, though, is that I was, a few months ago, looking to try my hand at optimizing ZBBlock a bit and send you a diff of my changes. Unfortunately, with it being as unreadable as it is, I can't invest the time in doing so because it'll take much too long to work out the code's flow. Classes burn up a bunch of time as is, and I've got a bunch of other projects to hammer out as well.

If you work out the readability problems and clean it up, let me know; I've got some great tools for profiling, debugging, and optimization, and I know a lot of optimization tricks that could help quite a bit.

Alex Kemp · 2011-06-05 6:11 pm

Obsidian wrote:

looking to try my hand at optimizing ZBBlock a bit

I humbly apologise.

qu.dawei · 2011-06-09 4:26 am

Thanks for this information, as it is just the kind of thing I have been looking for. Unfortunately, however, I can't access ZB Block because I am in China, and I've been caught out by an automatic blocking/honeypot registration system Zaphod has installed. Of course, I understand why this has
been done, but not everyone from within China is a forum spammer or hacker, and I'll see if I
can gain access to ZB Block by emailing Zaphod and seeing if I've given him enough information
for him to trust me enough to give me access.

I've recently had to suspend all registration attempts and access to my own forum because of the
increasing numbers of registration attempts that I knew with absolute certainty were from spammers
(I know this because of the nature of the forum that I am running which is exclusively for friends
and family relations of ours). I need some way of automatically dealing with such registration attempts
and the tools shown here would help me in that task.

#1 2010-08-31 4:51 pm

80legs web spider

#2 2010-08-31 4:56 pm

Re: 80legs web spider

#3 2010-08-31 5:05 pm

Re: 80legs web spider

#4 2010-08-31 5:19 pm

Re: 80legs web spider

#5 2010-08-31 5:51 pm

Re: 80legs web spider

#6 2010-08-31 6:08 pm

Re: 80legs web spider

#7 2010-08-31 6:11 pm

Re: 80legs web spider

#8 2010-08-31 6:30 pm

Re: 80legs web spider

#9 2010-08-31 7:31 pm

Re: 80legs web spider

#10 2010-09-01 12:21 am

Re: 80legs web spider

#11 2010-09-03 11:21 am

Re: 80legs web spider

#12 2010-09-03 11:32 am

Re: 80legs web spider

#13 2010-09-14 9:52 am

Re: 80legs web spider

#14 2011-06-03 12:16 am

Re: 80legs web spider

#15 2011-06-03 1:24 am

Re: 80legs web spider

#16 2011-06-03 4:57 pm

Re: 80legs web spider

#17 2011-06-04 2:36 am

Re: 80legs web spider

#18 2011-06-04 5:20 am

Re: 80legs web spider

#19 2011-06-05 3:03 am

Re: 80legs web spider

#20 2011-06-05 4:27 am

Re: 80legs web spider

#21 2011-06-05 1:44 pm

Re: 80legs web spider

#22 2011-06-05 4:51 pm

Re: 80legs web spider

#23 2011-06-05 6:11 pm

Re: 80legs web spider

#24 2011-06-09 4:26 am

Re: 80legs web spider

Board footer