Separate varnishncsa logs per domain

Tested in ubuntu 16 / Varnish 4.1.9

Here a init.d script it starts a daemon per domain using varnishncsa:

#!/bin/sh                                                                                                                                                                                                  
                                                                                                                                                                                                            
### BEGIN INIT INFO                                                                                                                                                                                         
# Provides:          vhostlog
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts vhostlog service
# Description:       starts vhostlog service
### END INIT INFO

LogsPath=/var/log/vhostlog

case "$1" in
    start)

        while read domain; do
            varnishncsa -D -q 'ReqHeader:Host ~ "^(www\.)?'$domain'$"' \
	        -a -w $LogsPath/$domain-access.log \
	        -F '%h %l %u %t "%m %U %H" %s %b "%{Referer}i" "%{User-agent}i"'

        done < /path/to/domains-list.txt
                
        ;;
    stop)

        killall varnishncsa
        sleep 3

        ;;
    restart)

        $0 stop
        $0 start

        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

The content of file /path/to/domains-list.txt could be like this:

domain1.com
domain2.com
domain-test.org
mybeautifuldomain.net

If varnish is behind another proxy (like nginx to serve SSL for example) you can change %h by %{X-Forwarded-For}i or %{X-Real-IP}i.

Once created update it in init.d:

~ $ update-rc.d vhostlog defaults

An finally start it:

~ $ /etc/init.d/vhostlog start

Varnish SSL redirect

Tested in Ubuntu 14 / Debian 8 / Varnish 4.1

If you have configured nginx as SSL proxy for varnish, you could be interested in redirecting requests from HTTP to HTTPS. We are going to suppose this configuration, so first in VCL recv, add this code:

sub vcl_recv {
    ...
    if (req.http.X-Forwarded-Proto !~ "(?i)https") {
        return (synth(750, ""));
    }
    ...
}

And then in VCL synth:

sub vcl_synth {
    ...
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "https://domain.com" + req.url;
    }
    ...
}

Finally reload varnish:

~ $ /etc/init.d/varnish reload

Varnish + SSL + WordPress

Tested in Ubuntu 14 / Varnish 4 / Nginx 1.11 / Apache 2.4

Using nginx as a proxy is the easiest and powerfull method to use SSL on a Varnish scenario, all incoming SSL traffic on 443 port will be redirected by nginx to varnish on port 80. Schema would be this:

Nginx(ssl) -> Varnish(caching) -> Apache|Nginx(backend) -> WordPress(app)

We assume that varnish is runnig and caching requests to the backend, so let’s go to install nginx:

~ $ apt-get install nginx

Now you have to create a virtual host file with the SSL and proxy parameters:

server {
        listen 443 ssl;

        server_name domain.com;
        ssl_certificate /etc/ssl/certs/domain.com.pem;
        ssl_certificate_key /etc/ssl/private/domain.com.key;

        location / {
                proxy_pass http://127.0.0.1:80;
                proxy_redirect off;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Forwarded-Port 443;
                proxy_set_header Host $host;
        }

}

Be sure nginx load this file, you can create it in /etc/nginx/conf.d directory with *.conf extension. Add this, if not exist, to the end of /etc/nginx/nginx.conf file inside the http block:

include /etc/nginx/conf.d/*.conf;

You can install a Let’s encrypt certificate or generate one self-signed.

Now restart nginx:

~ $ /etc/init.d/nginx restart

If you try to load domain.com via https probably you will see errors on load style sheets, images, even on secondary pages. This happens because wodpress doesn’t know that the connection is HTTPS, and internally try to serve content in plain HTTP.

To solve it, you have to tell the backend that changes to HTTPS if connection is originated in HTTPS.

Nginx as backend

Configure the HTTPS fastcgi parameter:

~ $ echo "fastcgi_param HTTPS $wordpress_https;" >> /etc/nginx/fastcgi_params

In /etc/nginx/nginx.conf add this to the http block:

map $http_x_forwarded_proto $wordpress_https {
       https on;
}

And restart nginx:

~ $ /etc/init.d/nginx restart

Apache as backend

Be sure apache has loaded mod_setenvif module first. Then create the config file domain.com.conf in /etc/apache2/conf-available/ with the content:

SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on

Create the symlink and restart apache:

~ $ ln -s /etc/apache2/conf-available/domain.com.conf /etc/apache2/conf-enable/domain.com.conf
~ $ /etc/init.d/apache2 restart

Now your wordpress should be loading correctly.

Varnish config for WordPress

Tested in Varnish 4

This is a example of configuration that you can use for caching wordpress:

sub vcl_recv {

        ## GENERAL CONFIG ##

        # Normalize host header
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Normalize Accept-Encoding header and compression
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                        unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Remove # at the end of URL
        if (req.url ~ "\#") {
                set req.url = regsub(req.url, "\#.*$", "");
        }

        # Remove ? at the end of URL
        if (req.url ~ "\?$") {
                set req.url = regsub(req.url, "\?$", "");
        }

        # Remove cookies with blanks
        if (req.http.cookie ~ "^\s*$") {
                unset req.http.cookie;
        }

	# Remove cookies for several extensions
        if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Remove cookies with only spaces
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Don't cache POST request
        if (req.http.Authorization || req.method == "POST") {
                return (pass);
        }

        ## WORDPRESS SPECIFIC CONFIG ##

	# Don't cache the RSS feed
        if (req.url ~ "/feed") {
                return (pass);
        }

        # Don't cache admin/login
        if (req.url ~ "/wp-(login|admin)") {
                return (pass);
        }

         # Don't cache WooCommerce
        if (req.url ~ "/(cart|my-account|checkout|addons|/?add-to-cart=)") {
                return (pass);
        }

	# Don't cache searchs
        if ( req.url ~ "\?s=" ){
                return (pass);
        }

        # Remove several cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

        # Don't cache wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
                return (pass);
        }

        ## RETURN ##

        # Cache the rest
        return (hash);

}

GeoIP Location in Varnish

Tested on Ubuntu 14 / Varnish 4

First install packages and libraries needed, you must use the varnish official repositories:

~ $ sudo apt-get install varnish varnish-dev git-core libgeoip-dev apt-transport-https libtool python-docutils automake make

Then download geoip vmod and compile it:

~ $ cd /usr/src/
~ $ git clone https://github.com/varnish/libvmod-geoip
~ $ cd libvmod-geoip
~ $ ./autogen.sh
~ $ ./configure
~ $ make
~ $ make install

Vmod will be installed in /usr/lib/varnish/vmods/.

TIP: The geoip database from repositories is a little bit outdated, so you can download the MaxMind free database to get better results:

~ $ cd /usr/share/GeoIP/
~ $ mv GeoIP.dat GeoIP.dat.old
~ $ wget -O GeoIP.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
~ $ gunzip GeoIP.dat.gz

Now in order to use its functions, first import the module in default.vcl file:

import geoip;

Then configure vmod geoip in varnish for blocking by country, for example if you want block China and Russia, in vcl_recv add first:

set req.http.X-Country-Code = geoip.country_code("" + client.ip);

to set the country code, and then to ban the country(s):

if (req.http.X-Country-Code ~ "(CN|RU)" ) {
        return (synth(403, "Forbidden"));
}

Now in vcl_synth, add something like this:

if (resp.status == 403) {
        synthetic( {"<!DOCTYPE html>
        <html>
        <head>
        <title>Forbidden</title> 
        </head>
        <body>
        <h1>Forbidden</h1>
        </body>
        </html>
        "} ) 
};

Finally reload varnish:

~ $ /etc/init.d/varnish reload