Migrate all databases to remote server with mysqldump in one step – 1:1 migration

Here is quick one, last in this year :). So I had to move a lot of databases to another server, but problem was, that on source server there wasn’t enough disk available. Also it was migration from very old mysql version to mariadb so mysqldump is your friend. Mysqldump all databases was out of the question because of low disk space. Dumping each database on its own would take too long and too many effort. But you can create dump of database and import it on new server in the same step.

First, you’ll need list of all databases on your source server and create them on new server. If your mysqldump creates “create database”, then you don’t need to create them manually on new server. If you want, you can skip mysql and any other databases that you don’t wish to transfer with grep. Put list of databases in some file – databases.txt for example. But first, make shure that command bellow show all databases. It is also necessary that you can remote access to mysql from source server to new server.

Test list all databases (exclude unwanted ones):

[root@oldserver ~]# mysql -e 'show databases' | grep -v "|" | grep -v "Database\|information_schema\|mysql\|performance_schema"
database1
geekytuts
database2
database3
database4
database5

Then put list of databases in text file databasest.txt:

[root@oldserver ~]# mysql -e 'show databases' | grep -v "|" | grep -v "Database\|information_schema\|mysql\|performance_schema" > databases.txt

Then you can import database to remote server like this:

mysqldump -u root -ppassword --single-transaction --skip-lock-tables database1 | mysql -h 1.1.1.1 -u root -ppassword database1

If you want to import all/multi databases, then use database.txt that we created in first step with for loop:

for i in `cat databases.txt`; do mysqldump -u root -ppassword --single-transaction --skip-lock-tables $i | mysql -h 1.1.1.1 -u root -ppassword $i; done

Bonus: If you need to create all databases listed in databases.txt on new server manualy, then you can also create all of them in one step. Use databases.txt on new server. If your mysqldump creates “create database” also, then you can skip this step.

[root@newserver ~]# for i in `cat databases.txt`; do mysql -u root -ppassword -e create database $i; done

Hope this helps someone.

Happy new year!

Permanent block ratelimited user with Rspamd and fail2ban

This one was a little tricky. I had few mail servers with a lot of accounts. I setup rspamd instance in proxy mode. Then I called rspamd on every mail server with postfix milter. Rspamd works beautifully, ratelimiting is very useful too. But I in case of abusive mail sender, I wanted to permanently block IP from which spam originated. You can’t permanently block IPs with rspamd because ratelimit module can’t add IP address to some file.

So Fail2ban came to mind. I setup fail2ban on my rspamd installation and create filter which watches rspamd log and wait for cases when ratelimit is triggered. When fail2ban counts 10 cases of triggered ratelimit, filter puts IP of ratelimited sender to special blacklist file (ip_blacklist_ratelimit.map) which is included in rspamd multimap  definition – permanent block. Spamer IP is blocked permanently this way. 

I had few cases when some users password was stolen and spam was sending. Fail2ban and rspamd sucsessfuly banned those IPs. I also created action which will notify administrator when fail2ban blocks IP.

Rspamd ratelimit config:


# limit outgoing authenticated users
user = { 
bucket = [ 
{ 
burst = 10; # capacity of 10 messages in bucket 
rate = "1 / 1min"; # leak 1 messages per minute 
}, 
{ 
burst = 100; # capacity of 100 messages in bucket 
rate = "30 / 60min"; # leak 30 messages per hour 
}] 
}
}

Rspamd multimap definition for blocking blacklisted IPs:


# block users exceeded ratelimits 5 times
IP_BLACKLIST_RATELIMIT {
type = "ip";
prefilter = "true";
map = "${LOCAL_CONFDIR}/local.d/maps/ip_blacklist_ratelimit.map";
action = "reject";
}

Fail2ban jail configuration:


[rspamd-ratelimit]
enabled = true
action = rspamd-banip
ratelimit-alert[name=Rspamd-ratelimit, dest=terminator@myemail.com]
backend = auto
filter = rspamd-ratelimit
logpath = /var/log/rspamd/rspamd.log
maxretry = 10
bantime = 3600

Fail2ban filter for rspamd – rspamd-ratelimit.conf:


# Fail2Ban filter for rspamd ratelimit
#
[INCLUDES]
before = common.conf
[Definition]
_daemon = rspamd_proxy
failregex = ^.*rspamd_proxy.*ip: .*?Ratelimit ".*?" exceeded

# Author: Igor Mazej

Fail2ban action for rspamd – rspamd-banip.conf:


#
# Author: Igor Mazej
#
#
[Definition]
actionstart = touch /etc/rspamd/local.d/maps/ip_blacklist_ratelimit.map
actionban = printf %%b "\n" >> /etc/rspamd/local.d/maps/ip_blacklist_ratelimit.map
actionunban = sed -i "//d" -i.backup /etc/rspamd/local.d/maps/ip_blacklist_ratelimit.map
[Init]

Directadmin – instal imagick (ImageMagick PHP module) with pecl – multi php

It’s amazingly simple. If you have multi PHP instances installed, go to bin directory of the one for which you want to install imagick module. In this example, I’ll install it on php version 7.3. This is CentOS 7 with Directadmin. You don’t need custombuild in this procedure.

Go to PHP bin directory:
[root@wutang4eva ~]# cd /usr/local/php73/bin/

Then just install imagick module likeå this:
[root@wutang4eva bin]# ./pecl install imagick

If php_ini is not set, then add extension to your php.ini manualy:
Add extension=imagick.so to /usr/local/php73/lib/php.ini

Restart php-fpm for php version - if you are using fpm of coruse:
systemctl restart php-fpm73.service

Check if imagick is installed:
[root@wutang4eva bin]# ./php73 -m | grep imag
imagick

If you want on other PHP versions, just do the same for each version you need imagick on.

I hope this helps.

Mac OSX/MacBook Pro – sound on headphones stops working. How to fix

This one is a little different and somehow weird. For some time now, I noticed that without obvious reason, my headphones stopped working on my MacBook Pro. If I switched to other sound output (internal speakers …), sound worked fine. After that I usually rebooted my laptop what became really annoying.

Then Dalibor showed me trick. Just open your Activity Monitor and find process “coreaudiod”. Select it and click on X mark on top left corner of the window. This will kill this process. Process will automatically start again. This should fix your issue.

Block wp-login and xmlrpc brute force attacks with CSF / cPanel

Another great counter attack to “flooders” on your WordPress installations. This time with CSF firewall. I had massive brute force attacks on WordPress installations on some cPanel server which were causing very high server loads.  Here is great way to block abusers with CSF firewall. Here is how.

First, create custom log from which CSF will be able to search for wp-login.php and xmlrpc.php requests. Edit your /etc/csf/csf.conf like bellow:

CUSTOM1_LOG = "/var/log/apache2/domlogs/*/*"

Because majority of those attacks are from some very well known country’s that are causing problems, you may want to white list country’s from which users shouldn’t be blocked. Add list of white list country’s in CC_IGNORE.

Then you must create custom functions for CSF so it will be able to block those attacks. Add this to your /usr/local/csf/bin/regex.custom.pm file. If it’s not there, create one. Then add this:

# XMLRPC
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /(\S+).*] "\w*(?:GET|POST) \/xmlrpc\.php.*" /)) {
return ("WP XMLPRC Attack",$1,"XMLRPC","5","80,443","1");
}

# WP-LOGINS
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /(\S+).*] "\w*(?:GET|POST) \/wp-login\.php.*" /)) {
return ("WP Login Attack",$1,"WPLOGIN","5","80,443","1");
}

Restart CSF and check if LFD is doing his new job. On success you should see something like this:

May 10 11:33:16 cp lfd[589350]: (WPLOGIN) WP Login Attack 4.4.4.4 (PL/Poland/s1.hekko.net.pl): 5 in the last 600 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]
May 10 11:33:36 cp lfd[589587]: (WPLOGIN) WP Login Attack 5.5.5.5 (TR/Turkey/5.5.5.5.linuxhosting.com.tr): 5 in the last 600 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]
May 10 11:34:24 cp lfd[590012]: (WPLOGIN) WP Login Attack 6.6.6.6 (DE/Germany/static.6.6.6.6.clients.your-server.de): 5 in the last 600 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]83247]: (WPLOGIN) WP Login Attack 7.7.7.7 (VN/Vietnam/-): 5 in the last 600 secs - *Blocked in csf* [LF_CUSTOMTRIGGER]
...

Requests for ignored country’s should look like this:

May 10 11:45:36 cp lfd[591718]: WP Login Attack 1.1.1.1 - ignored
May 10 11:45:41 cp lfd[591718]: WP Login Attack 2.2.2.2 - ignored
...

I hope this helps. 🙂

[kofi]

Postfix – OpenDKIM not signing outgoing email (not authenticated)

I had this weird issue. On CentOS 7 I set up mail server with Postfix, Dovecot, ClamAV, rspamd and OpenDKIM. Everything was working fine except all of my outgoing emails weren’t signed by DKIM.

In mail log this message was shown:

...
Mar 2 14:02:16 vps opendkim[10810]: 6FD942E30BB: not authenticated
Mar 2 14:02:16 vps opendkim[10810]: 6FD942E30BB: no signature data
...

After a lot of trying different configs an options, I googled answer on this forum. Apparently, opendkim thought that user wasn’t authenticated, but in reality, it was. This was due to Postfix configuration. There was missing milter macro. I added {auth_type} macro to milter_mail_macros. Then restarted Opendkim and Postfix and it started to work properly.

Now my postfix configuration (main.cf) for milters looks like this:

milter_protocol = 6
milter_default_action = accept
milter_connect_macros = j {daemon_name} v {if_name} _
milter_mail_macros="i {mail_addr} {client_addr} {client_name} {auth_type} {auth_authen}"
smtpd_milters = inet:localhost:11332 inet:localhost:8891 unix:/var/spool/postfix/clamav/clamav-milter.socket
non_smtpd_milters = $smtpd_milters

And emails are getting signed correctly:

...
Mar 2 14:15:56 vps opendkim[11513]: 8C5213E36D: DKIM-Signature field added (s=mail, d=domain.com)
...

CSF – whitelist user from SMTP_BLOCK

CSF features great option SMTP_BLOCK which block outgoing SMTP for all users except root, exim and mailman. I had a problem with one user which was using MailChimp as mass mailing within their application. Because of SMTP_BLOCK it wasn’t working. Disabling SMTP_BLOCK globally is not recommended, you can white list users for which you would like to allow sending.

Go to your CSF settings and find SMTP_ALLOWUSER. Then add user which should be allowed (users separated with coma). Don’t forget to restart CSF.

Get list of mass/multi domain redirects with CURL

I had large list of domains for which I had to check to which location are they pointing/redirecting. Curl is best option for this kind of work. To save some time, I wrote this simple one liner which will do that for you.

First, create txt file which will contain list of all domains that you want to check. For this example I will create domains.txt. 

Then, run this command – replace file name with yours.

> $ for i in `cat domains.txt`; do echo -n "$i -> "; curl -I -s -L -o /dev/null -w %{url_effective} -o /dev/null $i; echo "\t"; done

This will give you domain name with location to which it’s redirecting:

domain1.com -> https://www.domain1.com/sl 
domain1.de -> https://www.domain1.com/de 
domain2.si -> http://domain2.si/si 
example.com -> https://www.example.com/
lalala.es -> https://www. lalala.es/spain 
bash.com -> https://www.bash.com/i/love
...

Password protect Netdata with NGINX / Permission denied while connecting to upstream error

Netdata is great free tool for generating server statistics. By default it’s open for entire world on port 19999 – http://servername.com:19999. It is not a good idea to leave this open so everyone can your system statistics.

One way to limit access from where it is accessible is by editing netdata.conf and specify IPS in “allow connections from” variable.

[web]
allow connections from = ip's that are allowed to access>

There is no option to password protect it. This can be done with NGINX. You can create reverse proxy, so that nginx will serve content from netdata application. To make netdata accessible on subfolder of your hostname, eg. http://my.hostname.com/netdata, then create nginx configuration like bellow.

First generate password file for nginx:

htpasswd -c /etc/nginx/.htpasswd "username"

Then create or edit existing nginx configuration to something like this:

upstream netdata {
        server 127.0.0.1:19999;
        keepalive 64;
}

server {
     listen 443 ssl http2;
     server_name my.hostname.com;
     location = /netdata {
         return 301 /netdata/;
    }

    location ~ /netdata/(?.*) {

        auth_basic "Restricted Content";
        auth_basic_user_file /etc/nginx/.htpasswd;

        proxy_redirect off;
        proxy_set_header Host $host;   

        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
        proxy_pass http://netdata/$ndpath$is_args$args;

        gzip on;
        gzip_proxied any;
        gzip_types *;
   }

    error_log /var/log/nginx/error.log;
    access_log off;
    ....

Also, don’t forget to edit netdata.conf and change some variables. Make netdata accessible only from localhost (nginx):

[web]
bind to = 127.0.0.1
allow connections from = localhost
allow dashboard from = localhost

You should also allow connection to port 19999 only to local traffic (localhost).

Restart nginx and netdata, then try to access like: http(s)://my.hostname.com/netdata.

If you’re getting error like bellow in your nginx error log, than chances are that SELinux is active. Disable selinux or execute this command “setsebool -P httpd_can_network_connect true”.

[crit] 8411#0: *1 connect() to 127.0.0.1:19999 failed (13: Permission denied) while connecting to upstream, client: 8.8.8.8, server: my.hostname.com, request: "GET /netdata/ HTTP/1.1", upstream: "http://127.0.0.1:19999/", host: "my.hostname.com"

 

 

 

© 2024 geegkytuts.net
Hosted by SIEL


About author