When dealing with web servers where there are a lot of web sites, especially WordPresses, Joomlas etc., it is very common problem to dealing with flood/brute force attacks. One of most common for example, is generating massive requests on wp-login.php, or xmlrpc.php. With brute force, attackers goal is usually gaining access to administration. This is the simplest kind of method to gain access. Idea is very simple, attacker tries with a lot of different passwords and usernames, until it gets it right. Those operations of course, are automated by bots, scripts.
This can be very damaging for your server as it consumes a lot of memory. Every request means that someone just visited your website. When there is a script with bad intentions visiting your site, that means a lot of requests. Most modern web pages, every request like this, also makes database query. In most cases, server will become unresponsive, system will run out of memory, swap will fill up, mysql will stop responding.. This also means, that all websites on your server will stop working. In many cases, you’ll have to reboot your server to make it responsive again. Of course, there are systems that don’t allow this, like Cloud Linux with its LVE. One of great practices is to lock your administration to some static IP. There different ways.
But when you have a lot of users and you don’t have Cloud Linux and LVE available, and users are not locking their administrations, then its crucial that you detect this kind of behavior asap. I had issues on some older Directadmin servers which are running on older FreeBSD. With this combination, even firewall solution such as CSF is not possible. PF don’t offer any Directadmin integration. Of corse, brute force attacks are very common on any kind of servers which are hosting open source CMS systems such as WordPress, Joomla…
So I wrote a script in bash, Nagios plugin, that will detect when larger number of requests from specific IP will start to hit your server. Your Apache server status is great for this. It shows all current connections. So what this plugin does, is checking your server status for all requests and warns you, when larger number of requests are generated from specific IP – I already wrote that, didn’t I? 🙂 Of course, there are possible false positives, when some large or badly configured site will produce a massive amount of requests. But if you set your limits wisely, it can sure help you with brute force detection. When there are more than 40 requests from one IP address in your server status, you know that something is going on. Not necessarily, but probably.
I use this plugin with our Icinga monitoring system. It runs every 5 minutes or so. If there is warning or critical state, we get alert on our Telegram account instantly.
In the near future, I will also write a script that will check for brute force attacks and will block excessive IP addresses with firewall. It will work with PF, iptables, CSF. I will publish script here, on my blog where it’s finished. I think that running script with cron, every 2 minutes should do the trick. False positives are of course not excluded :-).
So here it is, at the bottom of this page. You can also download script from HERE and just wget it. I will explain it also. If you notice any errors, please let me know. At the beginning, allow access to your Apache /server-status to your Icinga/monitoring server or computer. It needs permission to server status of web server. How to do that, its described below.
- Create file check_bruteforce in libexec directory of your Nagios/Icinga installation. Make script executable also.
[root@monitor ~]# cd /usr/local/libexec/nagios [root@monitor ~]# vi check_bruteforce [root@monitor ~]# chmod +x check_bruteforce
You can set some default values if you want. You will be able to set limits and other values later when defining new Icinga/Nagios instance. NUM_DISPLAY_IPS is default value for how many top excessive IP’s do you want to display. MAX_CURL_EXEC_TIME will set curl timeout limit. It is defined in seconds. Other settings here should not be changed.
- Now you can try if its working. Run it from your command line. Here is how to display 5 top IP’s. Run -h for help.
igor@vincentvega# ./check_bruteforce -w3 -c 10 -H my.webserver.com -d 5 WARNING - Top 5 IPs: 6 connections from 93.103.175.85 3 connections from 212.28.22.1 1 connections from 85.10.33.143 1 connections from 78.45.128.8 1 connections from 78.11.88.211
- Now, create new command configuration so that you’ll be able to use it with Icinga. Open your commands.cfg configuration file of Icinga.
[root@monitor ~]# cd /usr/local/icinga/etc/objects [root@monitor ~]# vi commands.cfg
- Create new command definition like below. Curl timeout should be defined here. Also number of IP’s do display.
define command {
command_name check_bruteforce
command_line $USER1$/check_bruteforce -w $ARG1$ -c $ARG2$ -H $HOSTADDRESS$ -t 30 -d 3
}
- Go to your service configuration for server that you want to monitor brute force for and add something like this. Of course, change according to your configuration and needs. In this example, warning threshold is set to 50, critical threshold is set to 80, then we define URL on which server-status is available – myserverstatusdomain.com. Lastly, we define how many excessive IP’s to display in output. Set parameters according to your needs.
define service {
use generic-service
host_name my.webserver.com
service_description BRUTE FORCE
contact_groups geekytuts
check_command check_bruteforce!50!80!myserverstatusdomain.com!3
}
- You can restart your Icinga/Nagios. It is wise to check configuration first if it contains some kind of errors. You can check it like with /path/to/your/icinga/init/icinga checkconfig. If all ok, restart Icinga.
- At this point, you’ll probably getting error because your Icinga server is not able to check Apache server status of web server that you are trying to check. You should grant access to /server-status to your Icinga server so that it will be able to check for status. Connect to your web server and add IP of your Icinga server to your Apache configuration. If you already did this and its working fine, then great, good for you sir. 🙂
[root@webserver ~]# cd /etc/httpd/conf/extra/
[root@webserver /etc/httpd/conf/extra]# vi httpd-info.conf
# add Icinga server IP to the list. In this example IP is 123.123.123.123
SetHandler server-status
Order deny,allow
Deny from all
Allow from 213.161.21.184
Allow from 123.123.123.123
- Restart Apache. When refreshed, you should be able to see something like this on picture below.
Script:
#!/usr/bin/env bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
### set some values
# exit codes
EXIT_OK=0
EXIT_WARNING=1
EXIT_CRITICAL=2
EXIT_UNKNOWN=3
# version, author ...
PROGNAME=`basename $0`
VERSION="Version 1.0"
AUTHOR="2016, Igor Mazej (https://www.geekytuts.net/)"
# define some default values
NUM_DISPLAY_IPS=3
MAX_CURL_EXEC_TIME=30
# check if command installed function
exists()
{
command -v "$1" >/dev/null 2>&1
}
# check if curl is installed
if ! exists curl; then
echo "curl is not installed!"
exit $EXIT_UNKNOWN
else
CURL=`which curl`
fi
# print help function
function usage(){
echo -e '\n\n'
echo -e '**************************************************************************************'
echo -e '**************************************************************************************'
echo -e '\n'
echo $PROGNAME $VERSION
echo $AUTHOR
echo
echo -e 'With this script, you can check for excessive http requests from specific IP address.'
echo -e
echo -e 'OPTIONS:'
echo -e '\t -h Shows this help'
echo -e '\t -w sets the warning level'
echo -e '\t -c sets the critical level'
echo -e '\t -H sets hostname you wish to check. Must point to valid /server-status page'
echo -e '\t -d sets number of top excessive IPs to display (optional). Default value is 5.'
echo -e '\t -t sets curl timeout - in seconds (optional). Default value is 30 seconds.'
echo -e '\t -v display script version.'
echo -e '\n'
echo -e 'SAMPLE: check_bruteforce -w 10 -c 20 -H mywebsite.com [-t 30 -d 10]'
echo -e '\n'
echo -e '**************************************************************************************'
echo -e '**************************************************************************************'
echo -e '\n\n'
}
# print help if user provides no arguments
if [ $# -eq 0 ];
then
usage
exit 0
fi
# get data
while getopts “w:c:H:d:t:hv” opt; do
case $opt in
w)
WARNING_REQUESTS=$OPTARG
;;
c)
CRITICAL_REQUESTS=$OPTARG
;;
H)
HOSTNAME=$OPTARG
;;
d)
NUM_DISPLAY_IPS=$OPTARG
;;
t)
MAX_CURL_EXEC_TIME=$OPTARG
;;
h | --help)
if [ -z "$OPTARG" ];
then
usage;
exit $EXIT_OK
fi
;;
v)
if [ -z "$OPTARG" ];
then
echo $VERSION;
exit $EXIT_OK
fi
;;
\?)
echo "Invalid option: -$OPTARG" >&2
exit $EXIT_UNKNOWN
;;
:)
echo "Option -$OPTARG requires an argument." >&2
exit $EXIT_UNKNOWN
;;
esac
done
if [[ -z $CRITICAL_REQUESTS ]] || [[ -z $WARNING_REQUESTS ]] || [[ -z $HOSTNAME ]]
then
echo -e '\n'
echo -e "==========================================================================================================="
echo -e "\tMissing mandatory arguments! WARNING (-w), CRITICAL (-c) and HOSTNAME (-H) must be defined!"
echo -e "\t\t\t\t\tCheck manual for help (-h)"
echo -e "==========================================================================================================="
echo -e '\n'
exit $EXIT_UNKNOWN
fi
# get IPs from apache server status
NUM_REQUESTS=`$CURL --max-time $MAX_CURL_EXEC_TIME -s http://$HOSTNAME/server-status | grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" | sort -n | uniq -c | awk {'print $1 " connections from " $2'} | sort -nr | head -n$NUM_DISPLAY_IPS`
# check if there are any values in NUM_REQUESTS
if [[ -z $NUM_REQUESTS ]]
then
echo "UNKNOWN - No data from curl! Check your hostname!"
exit $EXIT_UNKNOWN
fi
# set output text
OUTPUT="Top $NUM_DISPLAY_IPS IPs:\t$NUM_REQUESTS"
# get number of requests from IP with most requests
NUM_TOP_REQUESTS=`printf "%s\n" "${NUM_REQUESTS[@]}" | awk {'print $1'} | sort -nr | head -n1`
if [ $NUM_TOP_REQUESTS -ge $CRITICAL_REQUESTS ]
then
echo -e "CRITICAL - $OUTPUT"
exit $EXIT_CRITICAL
elif [ $NUM_TOP_REQUESTS -le $CRITICAL_REQUESTS -a $NUM_TOP_REQUESTS -ge $WARNING_REQUESTS ]
then
echo -e "WARNING - $OUTPUT"
exit $EXIT_WARNING
else
echo -e "OK - $OUTPUT"
exit $EXIT_OK
fi