Rage Against the Shell – Linux tips and other things…

Compile varnish module vmod_vsthrottle

Posted on April 22, 2020 - May 23, 2020 by Mr. Reboot

Tested on Debian 9 / Varnish 5.0

Module vmod_vsthrottle is a varnish module to limit requests by unit time. It can be useful for APIs or as protection against an attack.

Varnish has grouped all official modules in a all-in-one package, so to compile vmod_vsthrottle is necessary compile all modules.

First install packages and libraries needed, of course you need varnish itslef:

~ $ apt-get install varnish libvarnishapi1 libvarnishapi-dev libtool python-docutils make pkg-config

Download varnish-modules from https://download.varnish-software.com/varnish-modules:

~ $ cd /usr/local/src/
~ $ wget https://download.varnish-software.com/varnish-modules/varnish-modules-0.15.0.tar.gz
~ $ tar xvzf varnish-modules-0.15.0.tar.gz
~ $ cd varnish-modules-0.15.0

Compile and install:

~ $ ./configure
~ $ make
~ $ make install

Check new modules were installed, included vmod_vsthrottle:

~ $ ls -l /usr/lib/x86_64-linux-gnu/varnish/vmods/
total 596
-rwxr-xr-x 1 root root  1019 Apr 22 13:15 libvmod_bodyaccess.la
-rwxr-xr-x 1 root root 51896 Apr 22 13:15 libvmod_bodyaccess.so
-rwxr-xr-x 1 root root   995 Apr 22 13:15 libvmod_cookie.la
-rwxr-xr-x 1 root root 56480 Apr 22 13:15 libvmod_cookie.so
-rw-r--r-- 1 root root 55208 Nov  9  2017 libvmod_directors.so
-rwxr-xr-x 1 root root   995 Apr 22 13:15 libvmod_header.la
-rwxr-xr-x 1 root root 52576 Apr 22 13:15 libvmod_header.so
-rwxr-xr-x 1 root root  1013 Apr 22 13:15 libvmod_saintmode.la
-rwxr-xr-x 1 root root 66048 Apr 22 13:15 libvmod_saintmode.so
-rw-r--r-- 1 root root 26560 Nov  9  2017 libvmod_std.so
-rwxr-xr-x 1 root root   977 Apr 22 13:15 libvmod_tcp.la
-rwxr-xr-x 1 root root 41912 Apr 22 13:15 libvmod_tcp.so
-rwxr-xr-x 1 root root   977 Apr 22 13:15 libvmod_var.la
-rwxr-xr-x 1 root root 63240 Apr 22 13:15 libvmod_var.so
-rwxr-xr-x 1 root root  1019 Apr 22 13:15 libvmod_vsthrottle.la
-rwxr-xr-x 1 root root 61328 Apr 22 13:15 libvmod_vsthrottle.so
-rwxr-xr-x 1 root root   983 Apr 22 13:15 libvmod_xkey.la
-rwxr-xr-x 1 root root 86000 Apr 22 13:15 libvmod_xkey.so

Now import module in /etc/varnish/default.vcl at the top of file:

import vsthrottle;

And add this to vcl_recv sub;

if (vsthrottle.is_denied(req.http.host, 15, 10s, 30s)) {
    return (synth(429, "Too Many Requests"));
}

Here we are using key “req.http.host”, it means will ban request by HTTP header host. More than 15 request in 10 seconds will be banned for 30 seconds.

You can use another key, for example X-Country-Code (using vmod_geoip), X-Forwarded-For, req.url, or several, for example:

if (vsthrottle.is_denied(req.http.host + X-Country-Code, 15, 10s, 10s)) {
    return (synth(429, "Too Many Requests"));
}

Bans requests for 10 seconds if ratio is more than 15 requests in 10 seconds when http header host and country code were the same.

Finally reload varnish:

~ $ /etc/init.d/varnish reload

Test it trying reach the limit:

~ $ curl -I -X GET http://domain.com

Use varnishncsa to see requests and code status:

~ $ varnishncsa -F '(Status:%s) %{X-Country-Code}i [%{Varnish:handling}x] %{X-Real-IP}i %m %{Host}i %U'

(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /
(Status:200) XX [pass] 10.10.10.2 GET domain.com /            
(Status:200) XX [pass] 10.10.10.2 GET domain.com /               
(Status:429) XX [synth] 10.10.10.2 GET domain.com /             
(Status:200) XY [pass] 10.10.10.3 GET domain.com /
(Status:200) XZ [pass] 10.10.10.4 GET domain.com /
(Status:429) XX [synth] 10.10.10.2 GET domain.com /
(Status:429) XX [synth] 10.10.10.2 GET domain.com /

SSH vpn with sshuttle

Posted on April 9, 2020 by Mr. Reboot

Tested in ubuntu 18

You can build your own vpn secure and fastly with sshuttle. This powerfull tool permits routing all traffic, trough a remote ssh server, by creating several rules on iptables.

Installing:

~ $ apt-get install sshuttle -y

Using:

~ $ sshuttle -v --dns -r user@myserver.domain.com 0/0

-v: verbose
–dns: dns traffic is sending by ssh tunnel
-r: remote server
0/0: destination to route, here all traffic

To redirect traffic for a specific destination:

~ $ sshuttle -v --dns -r user@myserver.domain.com 192.168.0.0/24

Disable swap in systemd

Posted on December 16, 2019 by Mr. Reboot

Tested in ubuntu 18 / debian 9

Usually you can disable the swap by this command:

~ $ swapoff -a

And then comment entry swap line in /etc/fstab file for making reboot persistent. But sometimes in systems with systemd this is not enough, steps would be these:

Get the swap service name in use:

~ $ systemctl --type swap

UNIT                                       LOAD   ACTIVE SUB    DESCRIPTION                  
dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap loaded active active /dev/mapper/ubuntu--vg-swap_1

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Stop service, note “‘” on start and end in service name:

~ $ systemctl stop 'dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap'

Mask service:

~ $ systemctl mask 'dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap'
Created symlink /etc/systemd/system/dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap → /dev/null.

Finally check service:

~ $ systemctl --type swap
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

If you want enable swap again:

~ $ systemctl unmask 'dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap'
~ $ systemctl start 'dev-mapper-ubuntu\x2d\x2dvg\x2dswap_1.swap'

Getting the parent process pid

Posted on October 12, 2018 by Mr. Reboot

Tested in debian 8

Sometimes can be useful to know the parent pid of a process for getting info or to kill it. You can use ps with ppid option:

Scenario:

root      1208  0.0 11.0 312388 113172 ?       Ss   Oct10   0:06 /usr/sbin/apache2 -k start
www-data  3915  0.0 11.1 1363796 113332 ?      Sl   06:25   0:05  \_ /usr/sbin/apache2 -k start
www-data  3916  0.0 10.9 1362304 112000 ?      Sl   06:25   0:03  \_ /usr/sbin/apache2 -k start

Getting de ppid knowing the child pid:

~ $ ps -O ppid= -p 3916
  PID       S TTY          TIME COMMAND
 3916  1208 S ?        00:00:03 /usr/sbin/apache2 -k start

Or short format with only pid:

~ $ ps -o ppid= -p 3916
 1208

Or knowing the child name:

~ $ ps -O ppid= -p $(pgrep apache2)
  PID       S TTY          TIME COMMAND
 1208     1 S ?        00:00:06 /usr/sbin/apache2 -k start
 3915  1208 S ?        00:00:05 /usr/sbin/apache2 -k start
 3916  1208 S ?        00:00:04 /usr/sbin/apache2 -k start

And viceversa, knowing the parent pid get pid from all childs:

~ $ ps --ppid=1208 -f
UID        PID  PPID  C STIME TTY          TIME CMD
www-data  3915  1208  0 06:25 ?        00:00:05 /usr/sbin/apache2 -k start
www-data  3916  1208  0 06:25 ?        00:00:04 /usr/sbin/apache2 -k start

Special characters in URL rewrite with mod_rewrite

Posted on October 10, 2018 - October 12, 2018 by Mr. Reboot

Tested in debian 8 / Apache 2.4.10

In apache enviroment with mod_rewrite, you can use the flag NE (no escape) for rewriting urls con special characters like #, ?, & … , example:

RewriteEngine On
RewriteRule ^(.*)$ "http://domain.com/#tag" [R=301,NC,L,NE]

R=301: type of redirection, 301 in this case
NC: no case or case insensitive
L: stop processing the rule set, like “break” in C

Separate varnishncsa logs per domain

Posted on February 11, 2018 - February 11, 2018 by Mr. Reboot

Tested in ubuntu 16 / Varnish 4.1.9

Here a init.d script it starts a daemon per domain using varnishncsa:

#!/bin/sh                                                                                                                                                                                                  
                                                                                                                                                                                                            
### BEGIN INIT INFO                                                                                                                                                                                         
# Provides:          vhostlog
# Required-Start:    $local_fs $remote_fs $network
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts vhostlog service
# Description:       starts vhostlog service
### END INIT INFO

LogsPath=/var/log/vhostlog

case "$1" in
    start)

        while read domain; do
            varnishncsa -D -q 'ReqHeader:Host ~ "^(www\.)?'$domain'$"' \
	        -a -w $LogsPath/$domain-access.log \
	        -F '%h %l %u %t "%m %U %H" %s %b "%{Referer}i" "%{User-agent}i"'

        done < /path/to/domains-list.txt
                
        ;;
    stop)

        killall varnishncsa
        sleep 3

        ;;
    restart)

        $0 stop
        $0 start

        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

The content of file /path/to/domains-list.txt could be like this:

domain1.com
domain2.com
domain-test.org
mybeautifuldomain.net

If varnish is behind another proxy (like nginx to serve SSL for example) you can change %h by %{X-Forwarded-For}i or %{X-Real-IP}i.

Once created update it in init.d:

~ $ update-rc.d vhostlog defaults

An finally start it:

~ $ /etc/init.d/vhostlog start

Snow in your shell

Posted on January 15, 2018 - January 15, 2018 by Mr. Reboot

Command to generate snow on your shell console, it’s cool:

~ $ clear;while :;do echo $LINES $COLUMNS $(($RANDOM%$COLUMNS)) $(printf "\u2744\n");sleep 0.1;done|gawk '{a[$3]=0;for(x in a) {o=a[x];a[x]=a[x]+1;printf "\033[%s;%sH ",o,x;printf "\033[%s;%sH%s \033[0;0H",a[x],x,$4;}}'

You will need gawk installed.

Source: http://climagic.org/coolstuff/let-it-snow.html (@climagic)

Suunto Moon Age App

Posted on July 7, 2017 by Mr. Reboot

Tested in Suunto Traverse

This app calculate moon age in days:

/* While in sport mode do this once per second */
MoonMonth = 29.53;
FirstNewMoonDayIn2000 = 6;

DaysFrom2000 = SUUNTO_DAYS_AFTER_1_1_2000 + (SUUNTO_TIME/86400);
DaysFromNewMoon = DaysFrom2000 - FirstNewMoonDayIn2000;
MoonAge = Suunto.mod(DaysFromNewMoon/MoonMonth,1) * MoonMonth;

postfix = "Days";
RESULT = MoonAge;

Glusterfs server and client at the same node

Posted on May 4, 2017 - May 23, 2020 by Mr. Reboot

Tested in Ubuntu 16 / Glusterfs 3.8

We’re going to configure a glusterfs cluster on two nodes with server and client on both hosts and without a dedicated partition or disk for storage.

First add name and IP address to the /etc/hosts file on both nodes, it’s important to configure glusterfs in a local network or to use a firewall to drop external traffic, for security reasons:

server01 10.10.0.1
server02 10.10.0.2

Then add glusterfs repositories, in this case the stable version was 3.8:

~ $ echo 'deb http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu xenial main' > /etc/apt/sources.list.d/gluster-glusterfs.list

Update and install needed packages:

~ $ apt-get upgrade
~ $ apt-get purge glusterfs-client glusterfs-server glusterfs-common

Start glusterfs daemon:

~ $ /etc/init.d/glusterfs-server start

Configure peers, in server01 type:

~ $ gluster peer probe server02 
~ $ gluster peer status

Or if you do it from server02, then:

~ $ gluster peer probe server01 
~ $ gluster peer status

List peers in the pool:

~ $ gluster pool list
UUID					Hostname   	State
bbc3443a-2433-4bba-25a1-0474ec77b571	server02	Connected 
df55a706-a32c-4d5b-a240-b29b6d16024b	localhost  	Connected

Now is the time to create a volume:

~ $ gluster volume create storage-volume replica 2 transport tcp server01:/storage-volume server02:/storage-volume force

gluster volume create: create a volume named storage-volume
replica 2: volume replication with two replicas, each node have a copy of all data
transport tcp: type protocol to use
server01:/storage-volume and server02:/storage-volume: node bricks
force: force create a volume in a root partition (root filesystem)

Start volume:

~ $ gluster volume start storage-volume

Show the volume status:

~ $ gluster volume status

Show the volume info:

~ $ gluster volume info

You can configure a lot of settings for tuning performance and security, for example permit traffic only between nodes:

~ $ gluster volume set storage-volume auth.allow 10.10.0.1,10.10.0.2

Or for improve IO performance (be carefully because it could be inconsistent):

~ $ gluster volume set storage-volume performance.flush-behind on

Now create a directory where mount the volume:

~ $ mkdir /mnt/dir-storage-volume

And finally mount in both nodes:

~ $ mount -t glusterfs 127.0.0.1:/storage-volume /mnt/dir-storage-volume

Now test the replication, writing in /mnt/dir-storage-volume directory on the first node and watch if changes are traslated to the second node, and vice-versa.

TIP: If you need add more bricks/nodes to extend the size of volume, first add the bricks and then extend replication with rebalance, remember we’re using two replicas:

~ $ gluster add-brick storage-volume replica 2 server03:/storage-volume server04:/storage-volume 
~ $ gluster rebalance storage-volume start
~ $ gluster rebalance storage-volume status

Boot on LVM root partition in Raspbian

Posted on March 8, 2017 - March 21, 2018 by Mr. Reboot

Tested in Raspberrypi 2 Model B / Raspbian Jessie

It’s possible booting on a LVM root partition using a external USB disk instead a SD card in Raspberrypi. Steps are the next (create lvm partitions is out of this post):

Check if your raspberrypi supports booting from initrd, is necessary for activate LVM on system boot:

~ $ zcat /proc/config.gz | grep INITRD
CONFIG_BLK_DEV_INITRD=y

If don’t exist /proc/config.gz load configs before:

~ $ modprobe configs

If result is “y” (usually) then create initrd file:

~ $ mkinitramfs -o /boot/initramfs.gz

In /boot/cmdline.txt file change the root partition replacing:

~ $ root=/dev/mmcblk0p2

with lvm disk using mapper designation, supposing vg0 as lvm group and lv01 as lvm volume:

~ $ root=/dev/mapper/vg0-lv01

You can add rootdelay=5 because interfaces can take a while to appear. At last add:

~ $ initramfs initramfs.gz

at the end of file /boot/config.txt. Finally reboot, with a little luck your rapsberrypi should boot on the new lvm root partition.