Varnish – Rage Against the Shell

Compile varnish module vmod_vsthrottle

Posted on April 22, 2020 - May 23, 2020 by Mr. Reboot

Tested on Debian 9 / Varnish 5.0

Module vmod_vsthrottle is a varnish module to limit requests by unit time. It can be useful for APIs or as protection against an attack.

Varnish has grouped all official modules in a all-in-one package, so to compile vmod_vsthrottle is necessary compile all modules.

First install packages and libraries needed, of course you need varnish itslef:

~ $ apt-get install varnish libvarnishapi1 libvarnishapi-dev libtool python-docutils make pkg-config

Download varnish-modules from https://download.varnish-software.com/varnish-modules:

~ $ cd /usr/local/src/
~ $ wget https://download.varnish-software.com/varnish-modules/varnish-modules-0.15.0.tar.gz
~ $ tar xvzf varnish-modules-0.15.0.tar.gz
~ $ cd varnish-modules-0.15.0

Compile and install:

~ $ ./configure
~ $ make
~ $ make install

Check new modules were installed, included vmod_vsthrottle:

~ $ ls -l /usr/lib/x86_64-linux-gnu/varnish/vmods/
total 596
-rwxr-xr-x 1 root root  1019 Apr 22 13:15 libvmod_bodyaccess.la
-rwxr-xr-x 1 root root 51896 Apr 22 13:15 libvmod_bodyaccess.so
-rwxr-xr-x 1 root root   995 Apr 22 13:15 libvmod_cookie.la
-rwxr-xr-x 1 root root 56480 Apr 22 13:15 libvmod_cookie.so
-rw-r--r-- 1 root root 55208 Nov  9  2017 libvmod_directors.so
-rwxr-xr-x 1 root root   995 Apr 22 13:15 libvmod_header.la
-rwxr-xr-x 1 root root 52576 Apr 22 13:15 libvmod_header.so
-rwxr-xr-x 1 root root  1013 Apr 22 13:15 libvmod_saintmode.la
-rwxr-xr-x 1 root root 66048 Apr 22 13:15 libvmod_saintmode.so
-rw-r--r-- 1 root root 26560 Nov  9  2017 libvmod_std.so
-rwxr-xr-x 1 root root   977 Apr 22 13:15 libvmod_tcp.la
-rwxr-xr-x 1 root root 41912 Apr 22 13:15 libvmod_tcp.so
-rwxr-xr-x 1 root root   977 Apr 22 13:15 libvmod_var.la
-rwxr-xr-x 1 root root 63240 Apr 22 13:15 libvmod_var.so
-rwxr-xr-x 1 root root  1019 Apr 22 13:15 libvmod_vsthrottle.la
-rwxr-xr-x 1 root root 61328 Apr 22 13:15 libvmod_vsthrottle.so
-rwxr-xr-x 1 root root   983 Apr 22 13:15 libvmod_xkey.la
-rwxr-xr-x 1 root root 86000 Apr 22 13:15 libvmod_xkey.so

Now import module in /etc/varnish/default.vcl at the top of file:

import vsthrottle;

And add this to vcl_recv sub;

if (vsthrottle.is_denied(req.http.host, 15, 10s, 30s)) {
    return (synth(429, "Too Many Requests"));
}

Here we are using key “req.http.host”, it means will ban request by HTTP header host. More than 15 request in 10 seconds will be banned for 30 seconds.

You can use another key, for example X-Country-Code (using vmod_geoip), X-Forwarded-For, req.url, or several, for example:

if (vsthrottle.is_denied(req.http.host + X-Country-Code, 15, 10s, 10s)) {
    return (synth(429, "Too Many Requests"));
}

Bans requests for 10 seconds if ratio is more than 15 requests in 10 seconds when http header host and country code were the same.

Finally reload varnish:

~ $ /etc/init.d/varnish reload

Test it trying reach the limit:

~ $ curl -I -X GET http://domain.com

Use varnishncsa to see requests and code status:

~ $ varnishncsa -F '(Status:%s) %{X-Country-Code}i [%{Varnish:handling}x] %{X-Real-IP}i %m %{Host}i %U'

(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /            
(Status:200) XX [pass] 10.10.10.2 GET domain.com /               
(Status:429) XX [synth] 10.10.10.2 GET domain.com /             
(Status:200) XY [pass] 10.10.10.3 GET domain.com /
(Status:200) XZ [pass] 10.10.10.4 GET domain.com /
(Status:429) XX [synth] 10.10.10.2 GET domain.com /
(Status:429) XX [synth] 10.10.10.2 GET domain.com /

Separate varnishncsa logs per domain

Posted on February 11, 2018 - February 11, 2018 by Mr. Reboot

Tested in ubuntu 16 / Varnish 4.1.9

Here a init.d script it starts a daemon per domain using varnishncsa:

#!/bin/sh                                                                                                                                                                                                  
                                                                                                                                                                                                            
### BEGIN INIT INFO                                                                                                                                                                                         
# Provides:          vhostlog
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts vhostlog service
# Description:       starts vhostlog service
### END INIT INFO

LogsPath=/var/log/vhostlog

case "$1" in
    start)

        while read domain; do
            varnishncsa -D -q 'ReqHeader:Host ~ "^(www\.)?'$domain'$"' \
	        -a -w $LogsPath/$domain-access.log \
	        -F '%h %l %u %t "%m %U %H" %s %b "%{Referer}i" "%{User-agent}i"'

        done < /path/to/domains-list.txt
                
        ;;
    stop)

        killall varnishncsa
        sleep 3

        ;;
    restart)

        $0 stop
        $0 start

        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

The content of file /path/to/domains-list.txt could be like this:

domain1.com
domain2.com
domain-test.org
mybeautifuldomain.net

If varnish is behind another proxy (like nginx to serve SSL for example) you can change %h by %{X-Forwarded-For}i or %{X-Real-IP}i.

Once created update it in init.d:

~ $ update-rc.d vhostlog defaults

An finally start it:

~ $ /etc/init.d/vhostlog start

Varnish SSL redirect

Posted on November 6, 2016 - November 6, 2016 by Mr. Reboot

Tested in Ubuntu 14 / Debian 8 / Varnish 4.1

If you have configured nginx as SSL proxy for varnish, you could be interested in redirecting requests from HTTP to HTTPS. We are going to suppose this configuration, so first in VCL recv, add this code:

sub vcl_recv {
    ...
    if (req.http.X-Forwarded-Proto !~ "(?i)https") {
        return (synth(750, ""));
    }
    ...
}

And then in VCL synth:

sub vcl_synth {
    ...
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "https://domain.com" + req.url;
    }
    ...
}

Finally reload varnish:

~ $ /etc/init.d/varnish reload

Varnish + SSL + WordPress

Posted on October 23, 2016 - October 23, 2016 by Mr. Reboot

Tested in Ubuntu 14 / Varnish 4 / Nginx 1.11 / Apache 2.4

Using nginx as a proxy is the easiest and powerfull method to use SSL on a Varnish scenario, all incoming SSL traffic on 443 port will be redirected by nginx to varnish on port 80. Schema would be this:

Nginx(ssl) -> Varnish(caching) -> Apache|Nginx(backend) -> WordPress(app)

We assume that varnish is runnig and caching requests to the backend, so let’s go to install nginx:

~ $ apt-get install nginx

Now you have to create a virtual host file with the SSL and proxy parameters:

server {
        listen 443 ssl;

        server_name domain.com;
        ssl_certificate /etc/ssl/certs/domain.com.pem;
        ssl_certificate_key /etc/ssl/private/domain.com.key;

        location / {
                proxy_pass http://127.0.0.1:80;
                proxy_redirect off;
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
                proxy_set_header X-Forwarded-Port 443;
                proxy_set_header Host $host;
        }

}

Be sure nginx load this file, you can create it in /etc/nginx/conf.d directory with *.conf extension. Add this, if not exist, to the end of /etc/nginx/nginx.conf file inside the http block:

include /etc/nginx/conf.d/*.conf;

You can install a Let’s encrypt certificate or generate one self-signed.

Now restart nginx:

~ $ /etc/init.d/nginx restart

If you try to load domain.com via https probably you will see errors on load style sheets, images, even on secondary pages. This happens because wodpress doesn’t know that the connection is HTTPS, and internally try to serve content in plain HTTP.

To solve it, you have to tell the backend that changes to HTTPS if connection is originated in HTTPS.

Nginx as backend

Configure the HTTPS fastcgi parameter:

~ $ echo "fastcgi_param HTTPS $wordpress_https;" >> /etc/nginx/fastcgi_params

In /etc/nginx/nginx.conf add this to the http block:

map $http_x_forwarded_proto $wordpress_https {
       https on;
}

And restart nginx:

~ $ /etc/init.d/nginx restart

Apache as backend

Be sure apache has loaded mod_setenvif module first. Then create the config file domain.com.conf in /etc/apache2/conf-available/ with the content:

SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on

Create the symlink and restart apache:

~ $ ln -s /etc/apache2/conf-available/domain.com.conf /etc/apache2/conf-enable/domain.com.conf
~ $ /etc/init.d/apache2 restart

Now your wordpress should be loading correctly.

Varnish config for WordPress

Posted on October 7, 2016 by Mr. Reboot

Tested in Varnish 4

This is a example of configuration that you can use for caching wordpress:

sub vcl_recv {

        ## GENERAL CONFIG ##

        # Normalize host header
        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Normalize Accept-Encoding header and compression
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                        unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Remove # at the end of URL
        if (req.url ~ "\#") {
                set req.url = regsub(req.url, "\#.*$", "");
        }

        # Remove ? at the end of URL
        if (req.url ~ "\?$") {
                set req.url = regsub(req.url, "\?$", "");
        }

        # Remove cookies with blanks
        if (req.http.cookie ~ "^\s*$") {
                unset req.http.cookie;
        }

	# Remove cookies for several extensions
        if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Remove cookies with only spaces
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Don't cache POST request
        if (req.http.Authorization || req.method == "POST") {
                return (pass);
        }

        ## WORDPRESS SPECIFIC CONFIG ##

	# Don't cache the RSS feed
        if (req.url ~ "/feed") {
                return (pass);
        }

        # Don't cache admin/login
        if (req.url ~ "/wp-(login|admin)") {
                return (pass);
        }

         # Don't cache WooCommerce
        if (req.url ~ "/(cart|my-account|checkout|addons|/?add-to-cart=)") {
                return (pass);
        }

	# Don't cache searchs
        if ( req.url ~ "\?s=" ){
                return (pass);
        }

        # Remove several cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

        # Don't cache wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
                return (pass);
        }

        ## RETURN ##

        # Cache the rest
        return (hash);

}

GeoIP Location in Varnish

Posted on May 23, 2016 - July 2, 2017 by Mr. Reboot

Tested on Ubuntu 14 / Varnish 4

First install packages and libraries needed, you must use the varnish official repositories:

~ $ sudo apt-get install varnish varnish-dev git-core libgeoip-dev apt-transport-https libtool python-docutils automake make

Then download geoip vmod and compile it:

~ $ cd /usr/src/
~ $ git clone https://github.com/varnish/libvmod-geoip
~ $ cd libvmod-geoip
~ $ ./autogen.sh
~ $ ./configure
~ $ make
~ $ make install

Vmod will be installed in /usr/lib/varnish/vmods/.

TIP: The geoip database from repositories is a little bit outdated, so you can download the MaxMind free database to get better results:

~ $ cd /usr/share/GeoIP/
~ $ mv GeoIP.dat GeoIP.dat.old
~ $ wget -O GeoIP.dat.gz http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
~ $ gunzip GeoIP.dat.gz

Now in order to use its functions, first import the module in default.vcl file:

import geoip;

Then configure vmod geoip in varnish for blocking by country, for example if you want block China and Russia, in vcl_recv add first:

set req.http.X-Country-Code = geoip.country_code("" + client.ip);

to set the country code, and then to ban the country(s):

if (req.http.X-Country-Code ~ "(CN|RU)" ) {
        return (synth(403, "Forbidden"));
}

Now in vcl_synth, add something like this:

if (resp.status == 403) {
        synthetic( {"<!DOCTYPE html>
        <html>
        <head>
        <title>Forbidden</title> 
        </head>
        <body>
        <h1>Forbidden</h1>
        </body>
        </html>
        "} ) 
};

Finally reload varnish:

~ $ /etc/init.d/varnish reload