September 2015 – Rage Against the Shell

NTP daemon status

Posted on September 27, 2015 - September 18, 2016 by Mr. Reboot

The ntpq utility is the tool to monitor the NTP daemon status. After configure NTP server you can use this tool to see the synchronization server status, and determine performance:

~ $ ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================

*217.114.59.66   75.12.230.181    2 u  127  128  377    8.786   -0.448   0.094
-193.225.118.163 228.143.95.23    2 u   70  128  377   36.129   -1.790   0.079
-46.22.223.220   129.242.4.241    2 u   52  128  377   40.330    3.975   0.055
+91.234.160.19   162.23.41.55     2 u   57  128  377   21.482   -0.364   0.093
 127.127.1.0     .LOCL.          10 l  37m   64    0    0.000    0.000   0.000

remote column:

* : custom time server source
# : source selected, distance exceeds maximum value
o : the PPS (Pulse Per Second) source if you have a PPS capable system and refclock
+ : candidate, it is considered a good source
– : discarded, quality is not good
x : false ticker, this one is considered to distribute bad time
blank : source discarded, failed sanity or high stratum

refid column:

The identification of the time source to which the remote machine is synced. May be a radio clock or another ntp server.

st column:

Stratum of the remote machine, 16 is “unsynchronized”, 0 is the best value. A stratum 0 could be a radio clock or the ntp servers private caesium clock.

t column:

l = local (such as a GPS, WWVB)
u = unicast (most common)
m = multicast
b = broadcast
– = netaddr

when column:

Seconds since the last poll of the remote machine.

poll column:

The polling interval in seconds.

reach column:

Connections to reach source, 377 means all connections got success.

delay column:

The time delay (in milliseconds) to communicate with the remote source.

offset column:

The offset (in milliseconds) between our time and that of the remote source.

jitter column:

The observed difference (in milliseconds) of time with the remote source.

MySQL innodb checksum mismatch

Posted on September 25, 2015 - September 25, 2016 by Mr. Reboot

Tested on Ubuntu 12 and 14 / Debian 7

When you find the error InnoDB: Error: checksum mismatch in data file ./ibdata1 (or ./ibdataX) at the mysql error log, it means that ibdata checksum is not correct and there are transactions that have not been completed and wrote to the ibdata, for example due to a low memory, oom killer, mysql service crash etc.

This could be the error at mysql error log:

...
InnoDB: Error: checksum mismatch in data file ./ibdata1
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB 
InnoDB: created in this failed attempt. InnoDB only wrote those 
InnoDB: files full of zeros, but did not yet use them in any way.  
InnoDB: But be careful: do not remove old data files which contain 
InnoDB: your precious data!
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
...

You can control the checksum, certainly it will show one or more errors:

~ $ innochecksum -d ibdata1

To solve it writing all changes to the ibdata file, first start mysql in recovery mode, at level 4. Add this to the my.cnf under [mysqld] tag:

innodb_force_recovery = 4

Start mysql:

~ $ /etc/init.d/mysql start

Now at the error log we can see:

...InnoDB: Log scan progressed past the checkpoint lsn 5294845868
InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the 
InnoDB: doublewrite buffer...
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 0.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Recovered the page from the doublewrite buffer.
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 371.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Recovered the page from the doublewrite buffer.
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 197.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Recovered the page from the doublewrite buffer.
InnoDB: Doing recovery: scanned up to log sequence number 5294846217
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Waiting for the background threads to start
InnoDB: 5.5.44 started; log sequence number 5294846217
InnoDB: !!! innodb_force_recovery is set to 4 !!!
...

It will make a data recovery. Now you have to disable innodb_force_recovery at my.cnf and kill mysql:

~ $ killall -9 mysqld

Usually mysql should start after kill, if not you can start it:

~ $ /etc/init.d/mysql start

At this time mysql should be already working fine!

Configure a NTP server

Posted on September 24, 2015 - August 14, 2016 by Mr. Reboot

S.O: Ubuntu 14.04 LTS

Install ntp daemon:

~ $ apt-get install ntp

Now modify the configuration file /etc/ntp.conf, first configure the servers which will synchronize, for example:

server 1.2.3.1 prefer
server 1.2.3.2
server 1.2.3.3
server 1.2.3.4

With the option prefer you can establish preferred servers, in this case server 1.2.3.1. It’s optional.

Add drift file and log file:

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log

The drift file contains the latest estimate of clock frequency error, and taking it as a reference, the NTP daemon will learn to adjust clock automatically.

Add localhost as a fallback server con a high stratum so that if none of our synchronization servers are working, NTP daemon can serve the local time (system clock time):

server 127.127.1.0
fudge 127.127.1.0 stratum 10

Stratum is a parameter used to synchronize the time based on distance. A server working at stratum 1 belongs to the class of best NTP servers available, because it has an accurate reference clock. So a stratum 10 server will have a low priority and usually will not be used for synchronize unless all other servers fail.

We want make a restrictive configuration so block all traffic by default:

restrict -4 default ignore
restrict -6 default ignore

And then permit the localhost;

restrict 127.0.0.1
restrict ::1

Synchronization servers:

restrict 1.2.3.1 kod notrap nomodify nopeer noquery
restrict 1.2.3.2 kod notrap nomodify nopeer noquery
restrict 1.2.3.3 kod notrap nomodify nopeer noquery
restrict 1.2.3.4 kod notrap nomodify nopeer noquery

And clients:

restrict 10.10.10.0 mask 255.255.0.0 nomodify notrap
restrict 192.168.172.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.0.10 nomodify notrap

The meaning of the restriction options is the next:

notrap: rejects packets not authenticated
nomodify: blocks access to ntpq and ntpdc queries which try to change the state of the server
nopeer: deny packets that try to making peering
noquery: rejects ntpq and ntpdc queries, time service is not affected

Finally restart the server:

~ $ /etc/init.d/ntp restart

TIP: if you want start service on IPv4 only, you can configure this at /etc/default/ntp file:

NTPD_OPTS='-4 -g'

Now at the client side you can use ntpdate to synchronize time. Install the packet:

~ $ apt-get install ntpdate

And create a cront task to execute ntpdate every 10 minutes:

*/10 * * * * /usr/sbin/ntpdate -u myntpserver.domain.com

Generate SSH key

Posted on September 5, 2015 - April 24, 2016 by Mr. Reboot

To generate the pair public/private keys type on the shell:

~ $ ssh-keygen -t rsa -b 2048

It ask you for a pass-phrase, it’s important choose a stronge pass-phrase, letters, numbers, symbols etc. If you accept the default path and name for the key it will create two files, the private key, id_rsa, and the public key, id_rsa.pub, in .ssh folder at your HOME:

~/.ssh/id_rsa
~/.ssh/id_rsa.pub

Check permissions, 600 for files and 700 for the .ssh directory.

Now just copy the content of id_rsa.pub and add it to the user home know_hosts file, on the server you want to connect you:

~ $ cat id_rsa.pub >> /root/.ssh/know_hosts

Finally add the private key identity to the authentication agent:

~ $ ssh-add

It will ask you for the pass-phrase, type it. By this way the identity will be stored and now when you try to connect to the server with the public key, you will enter safely without password:

~ $ ssh root@server

Clone disks

Posted on September 5, 2015 - May 31, 2016 by Mr. Reboot

To clone a entire disk you can use the dd command:

~ $ dd if=/dev/sda of=/dev/sdb bs=10240k conv=noerror,sync

Or dd_rescue, more reliable than dd:

~ $ dd_rescue -v /dev/sda /dev/sdb

Also you can use pv to watch the progress:

~ $ dd if=/dev/sda bs=10240k conv=noerror,sync | pv -c -b -r -t > /dev/sdb

About a partition table you can copy it from a disk to another:

~ $ sfdisk -d /dev/sda | sfdisk /dev/sdb

Or save to a file:

~ $ sfdisk -d /dev/sda > partition-table.output

And restore then:

~ $ sfdisk /dev/sda < partition-table.output

Extract info from a SSL cert

Posted on September 5, 2015 - August 8, 2017 by Mr. Reboot

All info from a CSR file:

~ $ openssl req -noout -text -in www.mydomainssl.com.csr

All info from a CRT file:

~ $ openssl x509 -noout -text -in www.mydomainssl.com.crt

Expiration date:

~ $ openssl x509 -noout -dates -in www.mydomainssl.com.csr

Verify the CSR file:

~ $ openssl req -verify -in www.mydomainssl.com.csr

Verify the KEY:

~ $ openssl rsa -check -in www.mydomainssl.com.key

Compare the md5 KEY, CSR and CRT to check if they match:

~ $ openssl rsa -noout -modulus -in www.mydomainssl.com.key | openssl md5
~ $ openssl req -noout -modulus -in www.mydomainssl.com.csr | openssl md5
~ $ openssl x509 -noout -modulus -in www.mydomainssl.com.crt | openssl md5

Extract secure HTTP info remotely:

~ $ openssl s_client -showcerts -connect www.mydomainssl.com:443

Extract secure SMTP info remotely:

~ $ openssl s_client -showcerts -connect www.mydomainssl.com:465

Real IP on nginx behind a proxy

Posted on September 3, 2015 - September 25, 2016 by Mr. Reboot

Tested on Debian 7 / NGINX 1.10

When a web server is working behind a proxy, ha-proxy, varnish, or any web server in proxy mode, by default in the log file, you see the IP of proxy server not the real client IP, this can be a trouble for making statistics for example.

Usually you can configure the proxy server for making forwarding of the client IP by X-Forwarded-For header, for example in varnish4 you can do it so:

req.http.X-Forwarded-For = client.ip;

But it’s not enough because is still necessary modify the configuration in the destination web server.

Nginx can be configured to show the client IP but for do it we need to compile from the sources. At this article the workaround is for Debian 7.

First, install the development libraries needed:

~ $ apt-get install libpcre3-dev libgeoip-dev libssl-dev libc6 libpcre3 zlib1g lsb-base

Second, download sources from the official site and unpack files, then run configure with the desirable options, for our case we need with-http_realip_module option:

~ $ ./configure --sbin-path=/usr/local/sbin --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_geoip_module --with-pcre-jit

Make:

~ $ make

And install:

~ $ make install

Finally add next configuration to nginx.conf file inside http block, where x.x.x.x is the proxy server IP:

set_real_ip_from x.x.x.x;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

And restart nginx:

~ $ /etc/init.d/nginx restart

LVM Encryption with Luks

Posted on September 1, 2015 - April 24, 2016 by Mr. Reboot

First create the logical volume:

~ $ lvcreate -L100G -n lv01 vg0

Now create the encrypted volume, you will be asked for the master password, it’s very important you save it safely.

~ $ cryptsetup -y -v luksFormat /dev/vg/lv01

Open the encrypted volume, is mapping to /dev/mapper:

~ $ cryptsetup luksOpen /dev/vg/lv01 clv01

Show the status:

~ $ cryptsetup -v status /dev/mapper/clv01
/dev/mapper/clv01 is active.
  type:    LUKS1
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  device:  /dev/mapper/vg-lv01
  offset:  4096 sectors
  size:    2147479552 sectors
  mode:    read/write
  Command successful.

To see the headers on the encrypted partition:

~ $ cryptsetup luksDump /dev/vg/lv01

It’s advisable writing zeros to the disk, be patient with a huge disk:

~ $ dd if=/dev/zero of=/dev/mapper/clv01

Format the file system with the desirable format:

~ $ mkfs.ext4 /dev/mapper/clv01

Mount device, for example in /mnt/DATA:

~ $ mount /dev/mapper/clv01 /mnt/DATA/

Unmount device safely:

~ $ umount /dev/mapper/clv01
~ $ cryptsetup luksClose /dev/mapper/clv01