Mar 162013
 

So one thing you learn pretty quick once you move into the cloud, is that what you normally would do to stop bots, rogue traffic, hackers, etc doesn’t quit work…even at the packet level.  What I mean is this, you have a web server sitting behind an aws load balancer and its under attack.  You are running linux.  First thing you do is set up an iptables rule to drop connections from that ip address.  The problem is, iptables never sees that ip address.  Iptables can’t look into packets.  Instead, it sees the load balancer ip address.  The ip address of the user is hidden in the x-forward-for.  So first thing you need to do is enable x-forward-for logging in your web server.  I will use nginx as an example:

Adding x-forward-for for nginx.conf.  In this example, the address space 10.0.0.0/8 is the address space used by amazon internal network.

 
http {
# added by ed wiget ref elb and displaying real ip
real_ip_header X-Forwarded-For;
set_real_ip_from 10.0.0.0/8;
# end added by ewiget
log_format main '$remote_addr - $remote_user $time_local '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"' ;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

Once you have that done, you can now reload nginx.  After reloading nginx, you will be able to see the real ip address in the nginx log for all requests instead of seeing the load balancer request.

Back to our story, for some reason you see rogue traffic, bot traffic, or some other traffic in your logs you need to deny access and you are running linux, most people would think block them by iptables.  Except, iptables only see’s the ip address of the load balancer and you can’t block the load balancer.  So, what do we do?  Luckily, you can block the request at the application level or the service level…..nginx example to follow:

There are two ways to do this, allow all and block specific ips or deny all and allow specific ips.

Allow All Example (except those specifically blocked):

Inside of nginx.conf in the global http section, you can do this (make sure you DO NOT copy and paste this exactly.  Replace 127.0.0.1 in the example with the ip address you wish to block).

set $allow true;
if ($http_x_forward_for ~ " ?127\.0\.0\.1$") {     
     set $allow false;
}
if ($allow = false) {    
     return 403; 
}

Reload nginx once you make these changes, and your problem ip address will now receive a 403 forbidden message for every request.

Deny All (except those specifically allowed)

Inside of nginx.conf in the global http section, you can do this (make sure you DO NOT copy and paste this exactly.  Replace 127.0.0.1 in the example with the ip address you wish to block).  Replace 10.0.0.1 with whatever servers you want to allow.  In this example we are saying, by default nobody is allowed, however we want to allow all traffic from 10.0.0.1 except for 127.0.0.1 which returns a 403 forbidden message.

set $allow false;
if ($http_x_forwarded_for ~ " ?10\.0\.0\.1?$") {
   set $allow true;
}
if ($http_x_forward_for ~ " ?127\.0\.0\.1$") {
   set $allow false;
}
if ($allow = false) {
   return 403;
}

Reload nginx for the change to take place.

One other status code that you may want to consider….however it is not recommended.  A 444 is a special status code which causes nginx to drop the connection without sending a response. This error code is dropping the connection between nginx and the load balancer — the load balancer is then left to decide what to return to the client.