Rage Against the Shell – Page 3 – Linux tips and other things…

Viewing apt-get stdout log

Posted on October 9, 2016 - October 15, 2016 by Mr. Reboot

You can view the apt-get history like if you were in front of the terminal, by visualizing the log /var/log/apt/term.log:

~ $ cat /var/log/apt/term.log

The output would be something like:

Setting up php5-gd (5.6.24+dfsg-0+deb8u1) ...
php5_invoke: Enable module gd for fpm SAPI
php5_invoke: Enable module gd for apache2 SAPI
php5_invoke: Enable module gd for cli SAPI
Setting up php5-mysqlnd (5.6.24+dfsg-0+deb8u1) ...
php5_invoke: Enable module mysqlnd for fpm SAPI
php5_invoke: Enable module mysqlnd for apache2 SAPI
php5_invoke: Enable module mysqlnd for cli SAPI
php5_invoke: Enable module mysql for fpm SAPI
php5_invoke: Enable module mysql for apache2 SAPI
php5_invoke: Enable module mysql for cli SAPI
php5_invoke: Enable module mysqli for fpm SAPI
php5_invoke: Enable module mysqli for apache2 SAPI
php5_invoke: Enable module mysqli for cli SAPI
php5_invoke: Enable module pdo_mysql for fpm SAPI
php5_invoke: Enable module pdo_mysql for apache2 SAPI
php5_invoke: Enable module pdo_mysql for cli SAPI
Setting up php5-dev (5.6.24+dfsg-0+deb8u1) ...
Setting up php5-fpm (5.6.24+dfsg-0+deb8u1) ...

Configuration file '/etc/init.d/php5-fpm'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** php5-fpm (Y/I/N/O/D/Z) [default=N] ?

Even in ncurses format:

apt-ncurses-screen

Varnish config for WordPress

Posted on October 7, 2016 by Mr. Reboot

Tested in Varnish 4

This is a example of configuration that you can use for caching wordpress:

sub vcl_recv {

        ## GENERAL CONFIG ##

        # Normalize host header
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Normalize Accept-Encoding header and compression
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                        unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Remove # at the end of URL
        if (req.url ~ "\#") {
                set req.url = regsub(req.url, "\#.*$", "");
        }

        # Remove ? at the end of URL
        if (req.url ~ "\?$") {
                set req.url = regsub(req.url, "\?$", "");
        }

        # Remove cookies with blanks
        if (req.http.cookie ~ "^\s*$") {
                unset req.http.cookie;
        }

	# Remove cookies for several extensions
        if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Remove cookies with only spaces
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Don't cache POST request
        if (req.http.Authorization || req.method == "POST") {
                return (pass);
        }

        ## WORDPRESS SPECIFIC CONFIG ##

	# Don't cache the RSS feed
        if (req.url ~ "/feed") {
                return (pass);
        }

        # Don't cache admin/login
        if (req.url ~ "/wp-(login|admin)") {
                return (pass);
        }

         # Don't cache WooCommerce
        if (req.url ~ "/(cart|my-account|checkout|addons|/?add-to-cart=)") {
                return (pass);
        }

	# Don't cache searchs
        if ( req.url ~ "\?s=" ){
                return (pass);
        }

        # Remove several cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

        # Don't cache wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
                return (pass);
        }

        ## RETURN ##

        # Cache the rest
        return (hash);

}

Unknown user in statoverride file

Posted on September 18, 2016 - September 18, 2016 by Mr. Reboot

If you get this error in Debian/Ubuntu on apt-get install/upgrade etc:

dpkg: unrecoverable fatal error, aborting:
 syntax error: unknown user 'user' in statoverride file

you can solve deleting ‘user’ in the statoverride file:

~ $ sed -i '/user/d' /var/lib/dpkg/statoverride

Now apt-get command should work fine.

MySQL load balancing with mysqlnd_ms plugin

Posted on September 18, 2016 - October 1, 2016 by Mr. Reboot

Tested on Debian 7 / PHP 5.4

In a mysql replication scenario you can balance sql queries sending write queries to the master server and read-only queries to the slaves using mysqlnd_ms plugin with mysqlnd php extension (mysql native driver). Also is possible in a master-master replication balance write queries between two or more masters.

First install php5-mysqlnd extension if is not installed yet, and other necessary packages.

~ $ apt-get install php5-mysqlnd php5-dev

Now you have to compile mysqlnd-ms plugin for mysqlnd, so download pecl extension from php.net:

~ $ wget http://pecl.php.net/get/mysqlnd_ms-1.5.2.tgz
~ $ tar xvzf mysqlnd_ms-1.5.2.tgz
~ $ cd mysqlnd_ms-1.5.2

Done this, configure:

~ $ phpize
~ $ ./configure --with-php-config=/usr/bin/php-config --enable-mysqlnd-ms

Compile:

~ $ make

And install:

~ $ make install

At last add extension mysqlnd_ms.so at the end of specific ini file, usually /etc/php5/mods-available/mysqlnd.ini:

extension=mysqlnd_ms.so

And enable it:

mysqlnd_ms.enable="on"

Now is time to configure the balancing type, two possible scenarios would be these:

One master, two slaves

Create /opt/php/mysqlnd_ms.json and add this:

{
   "mysqlbalancer": {
     "master": {
       "master1": {
         "host": "MASTER-IP",
         "port": "3306"
       }
     },
     "slave": {
        "slave1": {
         "host": "SLAVE1-IP",
         "port": "3306"
       }, 
	"slave2": {
         "host": "SLAVE2-IP",
         "port": "3306"
       }
     },
     "filters": {
           "random": {
              	"weights": {
                   "slave1":50,
                   "slave2":50
               }
           }
       },
     "master_on_write": 1,
     "failover": { "strategy": "loop_before_master", "remember_failed": true }
   }
 }

Add this to the file /etc/php5/mods-available/mysqlnd.ini:

mysqlnd_ms.config_file="/opt/php/mysqlnd_ms.json"

Here, write queries go to the master server and read queries go to the slaves randomly fifty-fifty (the same weight for both).

Parameter ‘master_on_write’ means that after writing into the master the next read query go to the master too, is more reliable because asynchronous mysql replication has delay.

With the option ‘loop_before_master’, if one slave fails, try another slave before making query into the master, and ‘remember_failed’ remembers if a node has been marked as failed discarding it.

Two or more masters

Create /opt/php/mysqlnd_ms.json and add this:

{
   "mysqlbalancer": {
     "master": {
       "master1": {
         "host": "MASTER1-IP",
         "port": "3306"
       },
       "master2": {
         "host": "MASTER2-IP",
         "port": "3306"
       }
     },
     "slave": { },
     "filters": { "roundrobin": [ ] },
     "failover": { "strategy": "loop_before_master", "remember_failed": true }
   }
 }

Add this to the file /etc/php5/mods-available/mysqlnd.ini:

mysqlnd_ms.config_file="/opt/php/mysqlnd_ms.json"
mysqlnd_ms.multi_master=1
mysqlnd_ms.disable_rw_split=1

Here, write and read queries go to both masters using roundrobin iterating into the master server list.

Option mysqlnd_ms.multi_master=1 enables the plugin to using several masters and mysqlnd_ms.disable_rw_split=1 disables separated read-write sentences because now masters work like a slaves too.

Once done configuration you have to restart services, if you use php-fpm do this:

~ $ /etc/init.d/php-fpm restart

if you use php module for apache then restart apache:

~ $ /etc/init.d/apache2 restart

Now you have to tell the web application that use this configuration, at the mysql connection string indicate the connector name defined above in the json file as HOST, example with pdo:

$conn = new PDO("mysql:host=mysqlbalancer;dbname=dbname", "username", "password");

If you use wordpress you have to modify wp-config.php changing DB_HOST var:

/** MySQL hostname */
define('DB_HOST', 'mysqlbalancer');

Finally, for security reasons you can force all mysql connections to use mysqlnd_ms plugin adding this option to the /opt/php/mysqlnd_ms.json file:

mysqlnd_ms.force_config_usage=1

Now if you try connecting to mysql using as HOST another name or IP address you will get an error like this:

PHP Warning:  PDO::__construct(): (mysqlnd_ms) Exclusive usage of configuration enforced but did not find the correct INI file section (localhost)
Connection failed: SQLSTATE[HY000] [2000] (mysqlnd_ms) Exclusive usage of configuration enforced but did not find the correct INI file section

+ Info: http://php.net/manual/en/mysqlnd-ms.configuration.php

MySQL Master-Master replication

Posted on July 24, 2016 - September 25, 2016 by Mr. Reboot

Tested on Ubuntu 14 / MySQL 5.5

We will configure mysql master-master replication on two nodes for all databases:

Node A: 1.1.1.1
Node B: 2.2.2.2

First install mysql in both nodes:

$ ~ apt-get install mysql-server mysql-client

In the node A configure this in my.cnf:

server-id                       = 1
log_bin                         = /var/lib/mysql/mysql-bin.log
log_bin_index                   = /var/lib/mysql/mysql-bin.log.index
relay_log                       = /var/lib/mysql/mysql-relay-bin
relay_log_index                 = /var/lib/mysql/mysql-relay-bin.index
binlog_format                   = ROW
slave_exec_mode                 = IDEMPOTENT
expire_logs_days                = 10
max_binlog_size                 = 100M
auto-increment-increment        = 2
auto-increment-offset           = 1

And this in the node B:

server-id                       = 2
log_bin                         = /var/lib/mysql/mysql-bin.log
log_bin_index                   = /var/lib/mysql/mysql-bin.log.index
relay_log                       = /var/lib/mysql/mysql-relay-bin
relay_log_index                 = /var/lib/mysql/mysql-relay-bin.index
binlog_format                   = ROW
slave_exec_mode                 = IDEMPOTENT
expire_logs_days                = 10
max_binlog_size                 = 100M
auto-increment-increment        = 2
auto-increment-offset           = 2

Explication:

server-id: the server identification, it must be different en each node
log_bin: path to the log binary file
log_bin_index: path to the log binary file index
relay_log: path to the log relay file
relay_log_index: path to the log relay file index
binlog_format: in this case is row based replication (ROW), all changes are replicated so is the safest form of replication. The disadvantage is that generates big update logs and more network traffic too.
slave_exec_mode: IDEMPOTENT is the best method for master-master replication, it causes suppression of duplicate-key and no-key-found errors.
expire_logs_days: log expiration in days
max_binlog_size: max size of binary logs
auto-increment-increment: the amount that the server will increment each time an auto incremented value is generated.
auto-increment-offset: the offset of increment. Example: auto-increment-increment=2 and auto-increment-offset, increment sequence will be 2, 4, 6 …

Restart mysql on both nodes:

$ ~ /etc/init.d/mysql restart

Create the replication user on both nodes:

mysql> CREATE USER 'repuser'@'%' identified by 'password'; 
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repuser'@'%';

Now in node A get the master status, remember file and position:

 mysql> SHOW MASTER STATUS;

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |      102 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Go to the node B and configure replication against the node A, using data before:

mysql> STOP SLAVE;
mysql> CHANGE MASTER TO MASTER_HOST='1.1.1.1', MASTER_USER='repuser', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=102, MASTER_PORT=3306;
mysql> START SLAVE;

In this node (B) get the master status, remember file and position:

mysql> SHOW MASTER STATUS;

+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 |      102 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

And back to the node A configure replication against node B, using data before:

mysql> STOP SLAVE;
mysql> CHANGE MASTER TO MASTER_HOST='2.2.2.2', MASTER_USER='repuser', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=102, MASTER_PORT=3306;
mysql> START SLAVE;

Check slave status on both nodes:

mysql> SHOW SLAVE STATUS\G

Postfix + OpenDMARC

Posted on June 18, 2016 - December 17, 2019 by Mr. Reboot

OpenDMARC is a spam control mechanism that complements SPF and DKIM checks, including reporting policies. In Debian/Ubuntu you can install opendmarc from repositories:

~ $ apt-get install opendmarc

On Debian wheezy you have to add backports to sources.list file before:

~ $ echo 'deb http://ftp.debian.org/debian wheezy-backports main contrib' >> /etc/apt/sources.list
~ $ apt-get update
~ $ apt-get install opendmarc

Change configuration en /etc/opendmarc.conf, take this as example:

AuthservID mail.server.com
PidFile /var/run/opendmarc.pid
RejectFailures false
Syslog true
TrustedAuthservIDs mail.server.com,mail2.server.com,mail3.server.com
UMask 0002
UserID opendmarc:opendmarc
IgnoreHosts /etc/opendmarc/ignore.hosts
HistoryFile /var/run/opendmarc/opendmarc.dat
SoftwareHeader true

Add hosts to /etc/opendmarc/ignore.hosts:

~ $ mkdir /etc/opendmarc
~ $ vim /etc/opendmarc/ignore.hosts
localhost
192.168.1.0/24

Configure port in /etc/default/opendmarc:

SOCKET="inet:83682@localhost"

And restart opendmarc:

~ $ /etc/init.d/opendmarc restart

Now configure postfix, add milters to existent milters (dkim) in /etc/postfix/main.cf:

milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:1234,inet:localhost:83682
non_smtpd_milters = inet:localhost:1234,inet:localhost:83682

Then restart:

~ $ /etc/init.d/postfix restart

Finally add _dmarc TXT dns record to your domain, example:

"v=DMARC1; p=none; pct=100; rua=mailto:[email protected]; ruf=mailto:[email protected]; fo=0; adkim=r; aspf=r"

Where:

v=DMARC1: dmarc version
p=none: means do nothing if dmarc check fails, is good for testing. You can use “p=quarantine” or “p=reject” instead.
pct=0: defines the percentage of mail to which policies are applied (0-100)
rua: address for sending reports
ruf: address for sending forensic reports
fo: reporting options, 0 generate a report if both SPF and DKIM checks failed
adkim: DKIM restrictive level (s=strict, r=relaxed)
aspf: SPF restrictive level (s=strict, r=relaxed)

Wait for the TTL refresh time and test if the record was created successfully:

~ $ dig +short txt _dmarc.server.com
"v=DMARC1\; p=none\; rua=mailto:[email protected]\; ruf=mailto:[email protected]\; fo=0\; adkim=r\; aspf=r"

Disable guest user in Ubuntu

Posted on June 4, 2016 - September 25, 2016 by Mr. Reboot

Tested on Ubuntu 14 and 16

You can disable guest user in Ubuntu, editing this file:

/usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf

and appending this to the end:

allow-guest=false

Then restart system:

~ $ sudo reboot

GeoIP Location in Varnish

Posted on May 23, 2016 - July 2, 2017 by Mr. Reboot

Tested on Ubuntu 14 / Varnish 4

First install packages and libraries needed, you must use the varnish official repositories:

~ $ sudo apt-get install varnish varnish-dev git-core libgeoip-dev apt-transport-https libtool python-docutils automake make

Then download geoip vmod and compile it:

~ $ cd /usr/src/
~ $ git clone https://github.com/varnish/libvmod-geoip
~ $ cd libvmod-geoip
~ $ ./autogen.sh
~ $ ./configure
~ $ make
~ $ make install

Vmod will be installed in /usr/lib/varnish/vmods/.

TIP: The geoip database from repositories is a little bit outdated, so you can download the MaxMind free database to get better results:

~ $ cd /usr/share/GeoIP/
~ $ mv GeoIP.dat GeoIP.dat.old
~ $ wget -O GeoIP.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
~ $ gunzip GeoIP.dat.gz

Now in order to use its functions, first import the module in default.vcl file:

import geoip;

Then configure vmod geoip in varnish for blocking by country, for example if you want block China and Russia, in vcl_recv add first:

set req.http.X-Country-Code = geoip.country_code("" + client.ip);

to set the country code, and then to ban the country(s):

if (req.http.X-Country-Code ~ "(CN|RU)" ) {
        return (synth(403, "Forbidden"));
}

Now in vcl_synth, add something like this:

if (resp.status == 403) {
        synthetic( {"<!DOCTYPE html>
        <html>
        <head>
        <title>Forbidden</title> 
        </head>
        <body>
        <h1>Forbidden</h1>
        </body>
        </html>
        "} ) 
};

Finally reload varnish:

~ $ /etc/init.d/varnish reload

Rename network interfaces in Ubuntu

Posted on May 19, 2016 - September 25, 2016 by Mr. Reboot

Tested on Ubuntu 16

From Ubuntu 15, network interfaces names are generated by systemd/udev using predictable names based in several policies and schemes like firmware, BIOS, interface’s MAC address, geographical location and others. Names like enp2s0 for example.

However it’s possible change the name, if you want retro-compatibility with old software or for scripting, to do it add net.ifnames=0 to the /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash net.ifnames=0"

Now make an update-grub:

~ $ sudo update-grub

And reboot:

~ $ sudo reboot

Interfaces now be called eth0, wlan0, eth1 etc…

Tuning ubuntu on Asus X series

Posted on May 19, 2016 - September 18, 2016 by Mr. Reboot

It’s probably you’ll have to tuning several things in Ubuntu 15/16 if you are an owner of a laptop ASUS X series.

If yo get this error on syslog and/or by stderr (a lot of records by second):

printk messages dropped pcieport 0000:1c.5 Receiver Error
pcieport 0000:00:1c.5 PCIe Bus Error: severity=Corrected, type=Physical Layer, id=00e5(Receiver ID)

add pci=nomsi to the /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=nomsi"

Also you’ll want control brightness via system keys, if don’t work, try adding acpi_osi=, acpi_backlight=native and intel_idle.max_cstate=1 to the /etc/default/grub file too:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=nomsi acpi_osi= acpi_backlight=native intel_idle.max_cstate=1"

Now make an update-grub:

~ $ sudo update-grub

At last, if you experiment disconnections on wireless mode check if your wlan driver is rtl8723be:

~ $ lspci | grep -i wireless
03:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8723BE PCIe Wireless Network Adapter

If yes create a configuration file en /etc/modprobe.d:

~ $ sudo touch /etc/modprobe.d/rtl8723be.conf

And add this:

~ $ sudo echo 'options rtl8723be fwlps=0' > /etc/modprobe.d/rtl8723be.conf

Finally reboot:

~ $ sudo reboot