October 2016 – Rage Against the Shell

Varnish + SSL + WordPress

Posted on October 23, 2016 - October 23, 2016 by Mr. Reboot

Tested in Ubuntu 14 / Varnish 4 / Nginx 1.11 / Apache 2.4

Using nginx as a proxy is the easiest and powerfull method to use SSL on a Varnish scenario, all incoming SSL traffic on 443 port will be redirected by nginx to varnish on port 80. Schema would be this:

Nginx(ssl) -> Varnish(caching) -> Apache|Nginx(backend) -> WordPress(app)

We assume that varnish is runnig and caching requests to the backend, so let’s go to install nginx:

~ $ apt-get install nginx

Now you have to create a virtual host file with the SSL and proxy parameters:

server {
        listen 443 ssl;

        server_name domain.com;
        ssl_certificate /etc/ssl/certs/domain.com.pem;
        ssl_certificate_key /etc/ssl/private/domain.com.key;

        location / {
                proxy_pass http://127.0.0.1:80;
                proxy_redirect off;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Forwarded-Port 443;
                proxy_set_header Host $host;
        }

}

Be sure nginx load this file, you can create it in /etc/nginx/conf.d directory with *.conf extension. Add this, if not exist, to the end of /etc/nginx/nginx.conf file inside the http block:

include /etc/nginx/conf.d/*.conf;

You can install a Let’s encrypt certificate or generate one self-signed.

Now restart nginx:

~ $ /etc/init.d/nginx restart

If you try to load domain.com via https probably you will see errors on load style sheets, images, even on secondary pages. This happens because wodpress doesn’t know that the connection is HTTPS, and internally try to serve content in plain HTTP.

To solve it, you have to tell the backend that changes to HTTPS if connection is originated in HTTPS.

Nginx as backend

Configure the HTTPS fastcgi parameter:

~ $ echo "fastcgi_param HTTPS $wordpress_https;" >> /etc/nginx/fastcgi_params

In /etc/nginx/nginx.conf add this to the http block:

map $http_x_forwarded_proto $wordpress_https {
       https on;
}

And restart nginx:

~ $ /etc/init.d/nginx restart

Apache as backend

Be sure apache has loaded mod_setenvif module first. Then create the config file domain.com.conf in /etc/apache2/conf-available/ with the content:

SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on

Create the symlink and restart apache:

~ $ ln -s /etc/apache2/conf-available/domain.com.conf /etc/apache2/conf-enable/domain.com.conf
~ $ /etc/init.d/apache2 restart

Now your wordpress should be loading correctly.

Error: page xxxx log sequence number yyyy is in the future

Posted on October 21, 2016 - October 23, 2016 by Mr. Reboot

Tested in Debian 8 / MySQL 5.5

If you’re getting this error on mysql error log:

InnoDB: Error: page 4352 log sequence number 12151412585
InnoDB: is in the future! Current system log sequence number 8204.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files

It’s due to a sequence mismatch in the InnoDB log, so it’s probable mysql doesn’t start. One thing you can do is restore a backup but, what happens if there is no backup?… Well, you can try to recover mysql using innodb-force-recovery.

First add this to the [mysqld] section in the my.cnf file:

innodb-force-recovery = 6

And then restart mysql:

~ $ /etc/init.d/mysql start

In the mysql error log you should get something like this:

InnoDB: Waiting for the background threads to start
InnoDB: 5.5.52 started; log sequence number 0
InnoDB: !!! innodb_force_recovery is set to 6 !!!
[Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
[Note]   - '127.0.0.1' resolves to '127.0.0.1';
[Note] Server socket created on IP: '127.0.0.1'.
[Note] Event Scheduler: Loaded 0 events
[Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.52'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306

If not, you are in trouble, look for a backup wherever… If yes, the next step is make a backup of all databases:

~ $ mysqldump -u root --all-databases > dump.sql

If dump ends successfully you are lucky. Now stop mysql:

~ $ /etc/init.d/mysql stop

Comment the innodb-force-recovery line from my.cnf file, and move ibdata*, ib_logfile* and all database folders, except mysql, to a temp dir:

~ $ cd /var/lib/mysql
~ $ mv ibdata* ib_logfile* database1 database2 (...) /tmp/

Start mysql, it will create new ibdata and ib_logfiles:

~ $ /etc/init.d/mysql start

In the error log:

InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
InnoDB: Waiting for the background threads to start
InnoDB: 5.5.52 started; log sequence number 0
[Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
[Note]   - '127.0.0.1' resolves to '127.0.0.1';
[Note] Server socket created on IP: '127.0.0.1'.
[Note] Event Scheduler: Loaded 0 events
[Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.52'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306

Mysql is working fine in a clean eviroment, the last step is restore the prior dump:

~ $ mysql -u root < dump.sql

At this time you should have all mysql databases, working like they were before crash.

Viewing apt-get stdout log

Posted on October 9, 2016 - October 15, 2016 by Mr. Reboot

You can view the apt-get history like if you were in front of the terminal, by visualizing the log /var/log/apt/term.log:

~ $ cat /var/log/apt/term.log

The output would be something like:

Setting up php5-gd (5.6.24+dfsg-0+deb8u1) ...
php5_invoke: Enable module gd for fpm SAPI
php5_invoke: Enable module gd for apache2 SAPI
php5_invoke: Enable module gd for cli SAPI
Setting up php5-mysqlnd (5.6.24+dfsg-0+deb8u1) ...
php5_invoke: Enable module mysqlnd for fpm SAPI
php5_invoke: Enable module mysqlnd for apache2 SAPI
php5_invoke: Enable module mysqlnd for cli SAPI
php5_invoke: Enable module mysql for fpm SAPI
php5_invoke: Enable module mysql for apache2 SAPI
php5_invoke: Enable module mysql for cli SAPI
php5_invoke: Enable module mysqli for fpm SAPI
php5_invoke: Enable module mysqli for apache2 SAPI
php5_invoke: Enable module mysqli for cli SAPI
php5_invoke: Enable module pdo_mysql for fpm SAPI
php5_invoke: Enable module pdo_mysql for apache2 SAPI
php5_invoke: Enable module pdo_mysql for cli SAPI
Setting up php5-dev (5.6.24+dfsg-0+deb8u1) ...
Setting up php5-fpm (5.6.24+dfsg-0+deb8u1) ...

Configuration file '/etc/init.d/php5-fpm'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** php5-fpm (Y/I/N/O/D/Z) [default=N] ?

Even in ncurses format:

apt-ncurses-screen

Varnish config for WordPress

Posted on October 7, 2016 by Mr. Reboot

Tested in Varnish 4

This is a example of configuration that you can use for caching wordpress:

sub vcl_recv {

        ## GENERAL CONFIG ##

        # Normalize host header
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Normalize Accept-Encoding header and compression
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                        unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Remove # at the end of URL
        if (req.url ~ "\#") {
                set req.url = regsub(req.url, "\#.*$", "");
        }

        # Remove ? at the end of URL
        if (req.url ~ "\?$") {
                set req.url = regsub(req.url, "\?$", "");
        }

        # Remove cookies with blanks
        if (req.http.cookie ~ "^\s*$") {
                unset req.http.cookie;
        }

	# Remove cookies for several extensions
        if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Remove cookies with only spaces
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Don't cache POST request
        if (req.http.Authorization || req.method == "POST") {
                return (pass);
        }

        ## WORDPRESS SPECIFIC CONFIG ##

	# Don't cache the RSS feed
        if (req.url ~ "/feed") {
                return (pass);
        }

        # Don't cache admin/login
        if (req.url ~ "/wp-(login|admin)") {
                return (pass);
        }

         # Don't cache WooCommerce
        if (req.url ~ "/(cart|my-account|checkout|addons|/?add-to-cart=)") {
                return (pass);
        }

	# Don't cache searchs
        if ( req.url ~ "\?s=" ){
                return (pass);
        }

        # Remove several cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

        # Don't cache wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
                return (pass);
        }

        ## RETURN ##

        # Cache the rest
        return (hash);

}