Hi everyone,
I've a LAMP webserver, with Apache 1.3 and PHP 4, MySQL 4 and Red Hat Enterprise 4 Update 5. Assuming the website is
formatting link
I receive about 20.000 unique users/day. Normally I have about 100 concurrent users and HTTP requests are like:
10.10.10.10 - - [16/Jun/2007:14:26:55 +0200] "GET / HTTP/1.1" 200
48711 "-" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/
20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:55 +0200] "GET
/stylesheet.css HTTP/
1.1" 200 8409 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET
/style2.css HTTP/
1.1" 200 1026 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET
/style3.css HTTP/
1.1" 200 513 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/logo2.gif HTTP/1.1" 200 4434 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu- edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/prova.gif HTTP/1.1" 200 1831 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu- edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:56 +0200] "GET /images/spacer2.gif HTTP/1.1" 200 43 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu-edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /userimgs/first.jpg HTTP/1.1" 200 21253 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu- edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /images/second.gif HTTP/1.1" 200 607 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu- edgy)"
10.10.10.10 - - [16/Jun/2007:14:26:57 +0200] "GET /images/third.gif HTTP/1.1" 200 197 "
formatting link
" "Mozilla/5.0 (X11; U; Linux i686; it; rv:1.8.1.4) Gecko/20060601 Firefox/2.0.0.4 (Ubuntu- edgy)"
The system load is 2.00 average (I know, it's high). The problem is the following. Sometimes I receive HTTP requests like this:
10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET
/page.php?id=1 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=2 HTTP/
1.1" 200 16174 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=3 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=4 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=5 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=6 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=7 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=8 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=9 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET
/page.php?id=10 HTTP/
1.1" 200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
or this:
10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:00 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16174 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
10.10.10.10 - - [15/Jun/2007:23:14:01 +0200] "GET /page.php HTTP/1.1"
200 16176 "
formatting link
" "Mozilla/4.0 (compatible; MSIE
7.0; Windows NT 5.1; InfoPath.1; .NET CLR 1.1.4322; .NET CLR
2.0.50727)"
that are malicious crawling attempts (first case) or DOS attacks (second case). In this cases my server load increase to 30-40 because every request is a query (or more than one because the PHP script query different tables) and I receive hundreds and hundreds of them. How can I detect and prevent this? I tried to use mod_evasive apache module, but it's based on request per second, so, for mod_evasive there isn't differences between a normal request (made up by a page and its resources like images, css, js, ecc) and a DOS attack (just page request) because the number of requests per second are the same (in my example the number of requests are 10).
Thanks to everyone and have a great weekend.