sysadmin

18 posts

Linux: query remote ntp (and show time difference)

Sometimes you just want to query a remote ntp server (and maybe see the difference between the two clocks). In this case you can use the tool sntp.
From the sntp man page:
The default is to write the estimated correct local date and time (i.e. not UTC) to the standard output….

use something like:
$ sntp at.pool.ntp.org
2018-04-19 11:36:26.538479 (-0100) -0.004752 +/- 0.023788 at.pool.ntp.org 86.59.28.10 s2 no-leap

OpenSSL Cheat Sheat

I know I know, there are plenty of openssl cheat sheets out there already((http://www.sslshopper.com/article-most-common-openssl-commands.html))((https://github.com/stanzgy/wiki/blob/master/network/openssl-self-signed-certs-cheatsheet.md)). But as I keep using googling it again and again to find the most useful openssl commands I decided to do my own – the first version of the blog entries is already 3 years old actually. Another reason for creating the list is that I remember things better when I am writing the down.

Show contents of a certificate (( https://knowledge.symantec.com/support/identity-protection-support/index?page=content&id=SO28771&actp=RSS&viewlocale=en_US ))

openssl x509 -fingerprint -sha256 -noout -text -in domain.com.crt

Show contents of a certificate request (CSR)

openssl req -text -noout -verify -in CSR.csr

OpenSSL HTTPS Client with SNI (( https://major.io/2012/02/07/using-openssls-s_client-command-with-web-servers-using-server-name-indication-sni/ ))

openssl s_client -connect roman.pertl.name:443 -servername roman.pertl.name

Generate a CSR to send to an external CA

openssl req -new -newkey rsa:2048 -sha256 -nodes -out roman.pertl.org.csr  -keyout roman.pertl.org.key  -subj "/C=US/ST=NY/L=NY/O=Roman Pertl/OU=Hostmaster/CN=roman.pertl.org"

Generate a Self-Signed SSL Certificate

openssl req -x509 -nodes -days 365 -newkey rsa:4096 -sha256 -keyout pertl.org.key -out pertl.org.crt -subj "/C=AT/ST=Korneuburg/L=Korneuburg/O=Pertl/OU=Hostmaster/CN=*.pertl.org"

Newer versions of browsers e.g. Chrome require to set the DNS alt attribute instead/additional of the CN field: ((https://serverfault.com/questions/845766/generating-a-self-signed-cert-with-openssl-that-works-in-chrome-58/845788))
((https://deliciousbrains.com/https-locally-without-browser-privacy-errors/))

openssl req -x509 -nodes -days 365 -newkey rsa:4096 -sha256 -reqexts SAN -extensions SAN -config <(cat /System/Library/OpenSSL/openssl.cnf <(printf '[SAN]\nsubjectAltName=DNS:*.pertl.org')) -keyout pertl.org.key -out pertl.org.crt -subj "/C=AT/ST=Korneuburg/L=Korneuburg/O=Pertl/OU=Hostmaster/CN=*.pertl.org"

Generate a Self-Signed CA Root Certificate

openssl req -nodes -newkey rsa:4096 -x509 -sha256 -days 3650 -keyout rootCA.key -reqexts v3_req -extensions v3_ca -out rootCA.crt -subj "/C=AT/ST=Korneuburg/L=Korneuburg/O=Pertl/OU=Hostmaster/CN=pertl CA"

Generate custom dh param file

openssl dhparam -out domain.at.dhparam 4096

View “raw” diff on Github

Viewing a “raw” diff in Github is easy – if you know how.

If you are viewing a commit on github the URL looks something like this:
https://github.com/voxpupuli/puppet-php/commit/50e1c1733a931dd3d9d21db8a4f584c9984100d1

Adding “.diff” to the URL will generate the raw diff for you which you can pipe to patch etc. The full URL is:
https://github.com/voxpupuli/puppet-php/commit/50e1c1733a931dd3d9d21db8a4f584c9984100d1.diff

Quickly create a (swap)file on linux

The traditional way for creating a linux swapfile would be using dd to create an empty file e.g.

dd if=/dev/zero of=/swapfile1 bs=1M count=2048

A faster way is to use “fallocate” e.g. (( http://www.cyberciti.biz/faq/ubuntu-linux-create-add-swap-file/ ))

fallocate -l 2G /swapfile1

Don’t forget the usually procedure for swapfiles:

chmod 0600 /swapfile1
chown root:root /swapfile1
mkswap /swapfile1
swapon /swapfile1
# edit /etc/fstab to add the swapfile during boot

Migration of OpenVZ Container to KVM Guest

This is a short tutorial how to migrate an OpenVZ container to a KVM Guest. Some ideas have been taken from other tutorials(( https://www.pither.com/simon/blog/2011/09/20/convert-an-openvz-vm-to-kvm )) (( http://blog.smile.fr/Migrate-your-openvz-containers-to-kvm-openstack )), the other half has been extracted from grml-debootstrap which can generate a KVM guest by using debootstrap.

 # form where to where migrate
 # don't forget the trailing / on the source!
 export SOURCE=/srv/vz/private/xxxxxxx/
 # an empty LVM volume!
 export DEST=/dev/mapper/vg0-myfirstvm

 # create some magic for grub/partion table
 echo 4 66 | /usr/share/grml-debootstrap/bootgrub.mksh -A | sudo dd of=${DEST} conv=notrunc
 sudo dd if=/dev/zero bs=1 conv=notrunc count=64 seek=446 of=${DEST}

 # this is a partition with a partion table on on its own
 sudo kpartx -av ${DEST}
 # create partition
 sudo parted -s "${DEST}" 'mkpart primary ext4 2M -1'
 # create filesystem
 sudo mkfs.ext4 "${DEST}p1"

 # mount it
 sudo mount "${DEST}p1" /mnt

 # not it's time to rsync your files to the destination folder (/mnt)
 sudo rsync -av --numeric-ids --stats --progress ${SOURCE}/ /mnt

 # mount some needed thins inside the KVM guest
 sudo mount -t proc none /mnt/proc
 sudo mount -t sysfs none /mnt/sys
 sudo mount --bind /dev /mnt/dev 

 # fix fstab
 echo `sudo blkid -o export  "${DEST}p1" |grep UUID `  /  ext4   defaults,noatime   0 0   | sudo tee -a /mnt/etc/fstab

 # install kernel + grub + acpi
 # IMPORTANT: during install you are asked to install grub - DO NOT install on any disks/partitions!
 sudo chroot /mnt aptitude install linux-image-amd64 linux-headers-amd64 busybox firmware-linux-free firmware-linux grub-pc acpid acpi-support-base
 # the last command installed and started acpid, so we need to stop it again
 sudo chroot /mnt service acpid stop
 # configure grub
 sudo chroot /mnt/ grub-mkimage -O i386-pc -p '(hd0,msdos1)/boot/grub' -o /tmp/core.img biosdisk part_msdos ext2
 # ATTEENTION: wheezy uses an old gurb and you need to copy it to boot/grub directly!
 sudo cp -rp /mnt/usr/lib/grub/i386-pc /mnt/boot/grub
 sudo dd if=/mnt/tmp/core.img of=$DEST conv=notrunc seek=4
 sudo chroot /mnt  update-grub

 # cleanup
 sudo umount /mnt/proc
 sudo umount /mnt/sys
 sudo umount /mnt/dev
 sudo umount /mnt
 sudo kpartx -d "${DEST}"

 # you may need to remount and rerun 'update-grub'
 # you should check /boot/grub/gurb.cfg within the virtual server that the linux kernel command arguments contain the correct root parameter (with a UUID)!
 # create the KVM VM with virt-install (see my previous post about KVM)

Update 2016/09/20: with some minor fixes and one missing command (copy grub/i386-pc directory)
Update 2016/10/04: I’ve found one more interesting post by ch: https://christian.hofstaedtler.name/blog/2012/07/openvz-to-kvm.html

Moving to KVM

Since our beloved OpenVZ virutalisation technology is not moving in a direction we are comfortable with, we are currently evaluation several virtualisation technologies. One the the possible options is KVM. This is not a full tutorial about KVM, there are many good tutorials already, e.g. ((https://www.lisenet.com/2016/getting-started-with-kvm-on-debian-jessie/)) or ((http://linuxnewbieguide.org/?p=1993)) or ((http://xmodulo.com/use-kvm-command-line-debian-ubuntu.html)) or ((http://wiki.libvirt.org/page/UbuntuKVMWalkthrough)) or ((http://www.cyberciti.biz/faq/how-to-install-kvm-on-ubuntu-linux-14-04/)), this is just a collection of some notes which I collected during the evaluation.

General Documentation

RedHat has some good KVM virtualisation documentation available at: https://access.redhat.com/documentation/en/red-hat-enterprise-linux/?version=7/
Debian also has some nice documentation: https://wiki.debian.org/KVM

Creating reproducable VMs for KVM


The easierst way is to use grml-debootstrap to create a new virtual maschine. We already have working netboot environment so we already have added some tuning to grml-debootstrap which makes it even easier.


sudo grml-debootstrap --hostname myfirstvm --vm --target /dev/mapper/vg0-myfirstvm
sudo virt-install --virt-type kvm --name=myfirstmv --vcpu=4 --ram=8192 \
--disk path=/dev/vg0/myfirstvm \
--os-variant=debianwheezy --cpuset=auto --network bridge=br0 --boot hd --vnc

virt-install has many more options(( http://www.techotopia.com/index.php/Installing_a_KVM_Guest_OS_from_the_Command-line_(virt-install) ))
You still need to setup the /etc/network/interfaces file inside your VM!

Examine VM configuration

virsh dumpxml client1

Serial Console for VM

You can use ‘virsh console clientvm’ to connect to the serial console of the virtual machine. In order to make use of it it, you need to activate the serial console in the VM((http://www.cyberciti.biz/faq/howto-setup-serial-console-on-debian-linux/)):

in order to see startup/shtudwon messages:

#/etc/default/grub
GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,19200n8'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1"

in order to be able to login: (for Debian Jessie with systemd)

systemctl start serial-getty@ttyS0.service
systemctl enable serial-getty@ttyS0.service

Mounting the virutal maschine disk

There are two options available: either with guestfish (install libguestfs-tools (( https://www.async.fi/2016/02/building-virtual-machines-with-vmbuilder/
)) or with virsh virt-edit (install libguestfs-tools ((https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#Accessing_data_on_guest_disk_images))


guestfish --rw --add /dev/vg0/myfistvm


virt-edit NameOfGuest /boot/grub/grub.conf

VM Remote Access with VNC

You need to specify a password in order to make it work with MacOSX buildin VNC client, otherwise the client won’t connect! You can add the password by editing the configuration (( http://www.cyberciti.biz/faq/linux-kvm-vnc-for-guest-machine/ ))

 <graphics type='vnc' port='-1' autoport='yes' passwd='mysuperduperpassword'/>

Support VM Shutdown

In order to support restart/shutdown from outside the KVM client, you need to install the following packages((http://serverfault.com/questions/549766/kvm-guest-with-acpi-installed-will-not-shutdown)):

– acpid
– acpi-support-base

Change VM parameters

Most parameters (e.g. RAM, CPU) cannot be changed during runtime. You can configure a maximal amount and a (lower) current amount. During runtime of a VM you can only allocate until the defined maximal amount. To increase the maximal value you need to shutdown the VM and change the configuration (( http://serverfault.com/questions/403561/change-amount-of-ram-and-cpu-cores-in-kvm#403671 )).

Lessions learned: CPU Placement

We have one VM which requires a lot of CPU usage (more cores than on one physical CPU). By default KVM seems to to limit one VM to one physical CPU, we need to adjust the settings directly in the XML to use all cpus(( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Manipulating_the_domain_xml-CPU_allocation.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Manipulating_the_domain_xml-CPU_tuning.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Managing_guest_virtual_machines_with_virsh-NUMA_node_management.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Overcommitting_with_KVM-Overcommitting_virtualized_CPUs.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/ch33s08.html )):

before:
<vcpu placement='static' cpuset='0-7'>16</vcpu>
after:
<vcpu placement='static' cpuset='0-15'>16</vcpu>

Summary

There are still many things to discover, rethink and consider for moving from OpenVZ to KVM, e.g. better resources planing as resources cannot be changed as easily as in OpenVZ. So there might be some updates to this post in the future. Stay tuned!

Running “backticks” commands on remote servers

Sometimes it’s necessary to run a complex command on a remote server witch also includes some “backticks”. Usually these commands are interpreted by the local shell so you need to use a little trick to force execution on the remote server:

ssh this.is.my.beautiful.server '( echo `echo "This Command is run on the remote server" ` )'

You need to use single quotes combined with brackets to use the backtick on the remote server.

Resetting Supermicor IPMI system

In case the IPMI system on a Supermicro system is unresponsive, but you are still able to log into the main server, you can issue the following command(s) to reset the IPMI:

# load the necessary modules (optional)
sudo modprobe ipmi_si
sudo modprobe ipmi_devintf
# reset the IPMI
sudo ipmitool mc reset cold
# remove all the modules
sudo rmmod ipmi_devintf
sudo rmmod ipmi_si

MacOSX: Manually restoring TimeMaschine Backup

In case you are a CLI junkie as myself and want to restore some files from a time maschine backup manually with the CLI (or using the finder), you will notice that the restored files cannot be changed. The restored files are copied with an ACL on the time machine backup witch prevents changes to those files. You need to remove the ACL from the restored files:
chmod -R -N restored-files/