Trendics Blog

Is someone establishing an excessive number of connections to your Linux server? Here are the commands I like to run to analyze connection usage.

First, let’s just store off the netstat output so we don’t have to retrieve it repeatedly…

echo "Saving current connections..."
netstat -nta > /tmp/netstat.txt

Next, let’s see the number of connections per IP address by extracting the IP address from the netstat output, counting the number of times each IP address was listed in the netstat output, and printing the top 10 (adjust head -10 appropriately if you’d like to see more than the top 10)…

echo "Number of connections per IP..."
cut -b 49-75 /tmp/netstat.txt | grep -o -P "\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b" | sort | uniq -c | sort -n -r -k 1,7 | head -10

If there is an excessive number of connections from a single IP, you may want to take action to block this IP.

Next, let’s see the number of connections in different states by extracting the state from the netstat output, and counting the number of times each state was listed in the netstat output…

echo "States of connections..."
cut -b 77-90 /tmp/netstat.txt | sort | uniq -c

The key reason to run this is to understand if you might be under a syn-flood attack. If you see an excessive number of connections in the SYN_RECV state, you may be under a syn-flood attack. It is normally unusual to see a high percentage of connections in a SYN_RECV state.

Next, let’s see the IP addresses generating connections currently in the SYN_RECV state by grepping the netstat output appropriately, counting the number of times each IP address was listed in the netstat output, and printing the top 10 (adjust head -10 appropriately if you’d like to see more than the top 10)…

echo "Number of SYN_RECV connections per IP..."
fgrep "SYN_RECV" /tmp/netstat.txt | cut -b 49-75 | cut -d ':' -f1 | sort | uniq -c | sort -n -r -k 1,7 | head -10

If the above command does not return any output, you simply do not have any connections in the SYN_RECV state which is good.

With all the above commands, we’ve analyzed the current connections to the server; however, this is not enough as you may not see IP addresses establishing lots of short connections. To analyze the number of new connections established to your server, you can execute this…

echo "Count number of new connection requests over the next 100 packets..."
time tcpdump -ns 200 -c 100 '(dst port http or dst port https) and tcp[13] & 2!=0' | grep -o -P '\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.\d{1,5}\s\>' | cut -d '.' -f 1-4 | sort | uniq -c | sort -n -r -k 1,7 | head -25

For this command, the -c parameter specified how many packets to analyze. Generally, I like to have this command run for about 30 seconds and I adjust the -c parameter to make it run over 30 seconds (very busy servers will need a much larger -c value and very light servers will need a much smaller -c value).

If the above command doesn’t ever return data, you have the -c parameter too high — basically the command blocks until the number of specified packets is received and your server has not yet received that number of packets.

In addition, the command above is only analyzing http/https connections; however, you can change the dst port http or dst port https to check other types of connections (consult the tcpdump documentation).

Also, it is useful to run all the commands above under normal conditions so you have a baseline to compare against when something abnormal occurs.

This entry was posted on Monday, July 28th, 2008 at 11:11 am and is filed under linux.You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.