Barrage of repeated requests for large files on my server from bots... what to do?

Hello all, first quickly I'm sorry if this is the wrong place, but it seemed the most appropriate usenet group.

My server is currently receiving a barrage of requests straight for the largest files (videos) on my web server, which was causing the server to become dreadfully slow to respond (at least a minute before it hints a response) and sometimes not even at all.

This is the 3rd day it's been going on.

I've been keeping track of stats from the bots for the last 16 hours...

- Requests were from 845 different IP addresses

- Nearly all the IP addresses are from China

- 30,377 requests for video files (not the video pages)

- 40 distinct video files were requested

- One host made 13,000 requests within those 40 files.

I'm running a dedicated server.

I've had a few similar "attacks" (or algorithm malfunction?) in the past, but it was only by 3 or 4 different IP addresses and so it was easy enough to just block them using iptables.

The only way I'm able to block the current "attack" and get these stats is that for some reason they all have the same referer:

formatting link
-- which I don't know about you but it doesn't even seem like a real website to me.

If the requests didn't have this referer, I would not have a way to block these requests and my server would be beaten to a pulp. Right now I'm redirecting any request with that referer to my logger script.

Any ideas of a better way to solve this once and for all? I'm passively blocking them, but they are still trying to jam the same requests down my server's throat. And I'm afraid that they'll get rid of the referer if I try any sort of active block rather than my passive method, and I'll be screwed.

The IP addresses also seem to be from several different hosts (judging from the whois data), so it's not exactly easy to inform them all...

An example of one of the bots from the access_log:

211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/ 439_poppy2606.wmv HTTP/1.1" 403 317 "
formatting link
""Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )" (Notice it's 403 because I'm blocking them myself)

Any tips at all?

Thanks in advance!!

- Richard

Reply to
richard k zad
Loading thread data ...

richard k zad wrote in news:c9f6d507-a64d-463b-95a7- snipped-for-privacy@w1g2000prd.googlegroups.com:

according to whois -h whois.apnic.net [quote] Please send abuse e-mail to remarks: snipped-for-privacy@gd.chinamobile.com remarks: Please send probe e-mail to remarks: snipped-for-privacy@gd.chinamobile.com [unquote]

Of course you could bloock all china.

05/07/08 04:24:16 Fast traceroute 211.139.255.10 Trace 211.139.255.10 ... .... 7 4.69.132.77 61ms 55ms 56ms TTL: 0 (ae- 3.ebr2.LosAngeles1.Level3.net ok) 8 4.69.137.22 55ms 53ms 52ms TTL: 0 (ae-72- 72.csw2.LosAngeles1.Level3.net ok) 9 4.68.20.69 51ms 57ms 52ms TTL: 0 (ae-23- 79.car3.LosAngeles1.Level3.net ok) 10 62.156.138.69 54ms 89ms 55ms TTL: 0 (No rDNS) 11 62.154.14.105 310ms 354ms 207ms TTL: 0 (lax- sa1.LAX.US.net.DTAG.DE ok) 12 62.154.5.205 210ms 208ms 208ms TTL: 0 (hkg- sa1.HKG.HK.net.DTAG.DE ok) 13 217.239.41.10 208ms 209ms 348ms TTL: 0 (No rDNS) 14 217.6.24.138 280ms 247ms 265ms TTL: 0 (No rDNS) 15 218.200.253.237 262ms 250ms 250ms TTL: 0 (No rDNS) 16 211.136.3.57 243ms 236ms 235ms TTL: 0 (No rDNS) 17 211.136.6.34 270ms 259ms 262ms TTL: 0 (No rDNS) 18 211.139.135.70 269ms 256ms 260ms TTL: 0 (No rDNS) 19 211.139.254.3 278ms 254ms 267ms TTL: 0 (No rDNS) 20 211.139.255.10 271ms 264ms * TTL:244 (No rDNS)

The lack of rDNS shows they are rather clueless. You might complain to HKG.HK.net.DTAG.DE about the bots It looks like HKG may be Hongcong and

Reply to
bz

Enter a subnet block for their providers IP range in your firewall so that no address in that range can reach your server.

You do have a proper firewall, right?

The following list works for me, and I'm inside the USA and don't deal with customers/users outside the USA, the entire 211.0.0.0/8 network is blocked in my list below:

Updated 2008MAR01

12.144.182.0/24 12.45.203.0/24 12.98.139.0/24 121.0.0.0/8 124.0.0.0/8 125.172.237.0/24 125.213.42.0/24 134.159.0.0/16 134.160.0.0/16 140.109.0.0/16 140.110.0.0/15 140.112.0.0/12 140.128.0.0/13 140.136.0.0/15 140.138.0.0/16 155.48.106.0/24 162.40.0.0/16 168.126.0.0/16 172.184.111.203 190.3.209.0/24 193.248.60.0/24 193.251.0.0/16 193.252.0.0/16 193.253.0.0/16 194.170.0.0/16 195.174.0.0/16 195.175.16.0/20 195.229.0.0/23 195.58.124.0/24 200.181.0.0/16 200.244.0.0/16 200.30.203.0/24 201.0.0.0/8 201.130.192.0/18 201.230.0.0/16 201.240.0.0/16 202.40.148.0-202.40.149.255 202.84.128.0-202.84.255.255 202.88.186.0/24 203.150.101.0/24 203.152.22.0/24 203.162.0.0-203.162.255.255 203.210.128.0-203.210.255.255 205.251.79.0/24 210.0.0.0/8 211.0.0.0/8 212.150.124.0/24 212.162.8.0/24 212.18.57.0/24 212.202.178.0/24 212.27.32.0-212.27.63.255 212.64.0.0/16 212.9.7.0/24 213.13.26.0/24 213.192.0.0-213.192.255.255 216.184.97.0/24 216.76.35.0/24 217.118.224.0-217.118.239.255 217.160.110.0/24 218.164.28.0/24 218.234.0.0-218.239.255.255 218.252.74.0/24 218.67.128.0-218.76.255.255 219.115.214.0/24 219.212.4.0/24 219.56.0.0/24 219.97.93.0/24 220.0.0.0/8 222.0.0.0/8 41.221.19.0/24 60.0.0.0/8 61.135.148.0/24 61.175.239.0/24 61.181.0.0/16 61.218.19.0/24 61.33.206.0/24 61.48.18.0/24 62.154.0.0/17 62.240.161.0-62.240.161.127 64.230.125.0/24 66.250.125.0/24 66.250.32.0/24 66.28.35.131 66.57.133.0/24 71.184.44.154 78.48.8.16 80.0.0.0/8 81.0.0.0/8 82.0.0.0/8 83.0.0.0/8 85.17.255.0-85.255.255.255 87.0.0.0/8 88.0.0.0/8 89.0.0.0/8 91.76.56.0/24
Reply to
Leythos

That's good, but most find that the abuse contacts are ignored.

[compton ~]$ zgrep -c CN APNIC.gz 1430 [compton ~]$

Don't forget Hong Kong and Macau (.hk and .mo, which adds 648 more networks) which are now special districts of China, and some people consider Taiwan (.tw with 397 more networks) as part of China for a total of 2475 address blocks, ranging from 200 /24s (255.255.255.0) on up to 4 /10s (255.192.0.0) - but hey, that's only 174,089,728 IPv4 addresses in total.

[compton ~]$ zgrep -E (CN|HK|MO|TW) APNIC.gz | cut -d' ' -f2 | cut -d'.' -f1 | sort -n | uniq -c | column 51 58 85 124 2 161 3 204 42 59 48 125 1 162 2 206 50 60 3 134 4 163 161 210 115 61 1 137 1 165 44 211 63 116 2 139 2 166 96 218 41 117 21 140 1 167 58 219 44 118 1 143 4 168 29 220 86 119 1 144 2 169 68 221 29 120 1 147 103 192 70 222 50 121 4 152 4 198 44 122 2 158 708 202 58 123 1 159 269 203 [compton ~]$

Translation: 51 address blocks in the 58.x.x.x range, 42 in the 59.x.x.x range, 50 in the 60.x.x.x range, and so on. And before you use a /8 rule, you should know that there are other countries using those blocks,

[compton ~]$ zgrep -h ' 58\\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c | column 1 AF 3 ID 2 NZ 4 TW 24 AU 3 IN 2 PH 1 VN 5 BD 30 JP 5 PK 43 CN 15 KR 6 SG 4 HK 4 MY 8 TH [compton ~]$ ^58^59 zgrep -h ' 59\\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c | column 8 AU 4 HK 6 KR 4 TW 1 BD 7 IN 1 PK 34 CN 16 JP 1 SG [compton ~]$

That's just two /8s. Do you know your ISO-3166 country codes?

and also makes it hard to use rDNS to identify Chinese blocks - but what else is new? This happens in a lot of countries.

And what IF ANYTHING does this have to do with the spam?

Why - they happen to be an intermediate step on YOUR route to China, but this is not as likely for the rest of us.

HKG is probably Hong Kong - a former British enclave on the coast of China around 22.2 degrees North, as LAX is Los Angeles. Your point?

Old guy

Reply to
Moe Trin

Since the same IP requests the file a bunch of times, maybe you could put a CGI interface infront of the file instead of serving the file directly then log the IPs in a database with a time stamp (or use the webserver logs or whatever) to see if multiple requests have been made in a short period of time. If the same IP is requesting it over and over again

while true do sleep 1 transmit byte done

Transmit the file really slowly so that the connection stays active, but does not use up your bandwidth.

You could also check the referer in the CGI and block anything which does not have a referer page at least from the same domain.

There is probably a way to do a better job with a rate limiting firewall rule. Maybe spamd on OpenBSD perhaps?

iptables has a rate limiting option

limit This module matches at a limited rate using a token bucket filter. A rule using this extension will match until this limit is reached (unless the =91!=92 flag is used). It can be used in combination with the LOG target to give limited logging, for example.

--limit rate Maximum average matching rate: specified as a number, with an optional =91/second=92, =91/minute=92, =91/hour=92, or =91/d= ay=92 suffix; the default is 3/hour.

--limit-burst number Maximum initial number of packets to match: this number gets recharged by one every time the limit specified above is not reached, up to this number; the default is 5.

Maybe you could create a rule limiting the average rate and the burst rate for these subnets. That way they can still connect but only use a fraction of your bandwidth.

73la.cn/u/egao/-- which I don't know about you but it
Reply to
spam

Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.