linux

9 posts

Linux: query remote ntp (and show time difference)

Sometimes you just want to query a remote ntp server (and maybe see the difference between the two clocks). In this case you can use the tool sntp.
From the sntp man page:
The default is to write the estimated correct local date and time (i.e. not UTC) to the standard output….

use something like:
$ sntp at.pool.ntp.org
2018-04-19 11:36:26.538479 (-0100) -0.004752 +/- 0.023788 at.pool.ntp.org 86.59.28.10 s2 no-leap

Quickly create a (swap)file on linux

The traditional way for creating a linux swapfile would be using dd to create an empty file e.g.

dd if=/dev/zero of=/swapfile1 bs=1M count=2048

A faster way is to use “fallocate” e.g. (( http://www.cyberciti.biz/faq/ubuntu-linux-create-add-swap-file/ ))

fallocate -l 2G /swapfile1

Don’t forget the usually procedure for swapfiles:

chmod 0600 /swapfile1
chown root:root /swapfile1
mkswap /swapfile1
swapon /swapfile1
# edit /etc/fstab to add the swapfile during boot

Moving to KVM

Since our beloved OpenVZ virutalisation technology is not moving in a direction we are comfortable with, we are currently evaluation several virtualisation technologies. One the the possible options is KVM. This is not a full tutorial about KVM, there are many good tutorials already, e.g. ((https://www.lisenet.com/2016/getting-started-with-kvm-on-debian-jessie/)) or ((http://linuxnewbieguide.org/?p=1993)) or ((http://xmodulo.com/use-kvm-command-line-debian-ubuntu.html)) or ((http://wiki.libvirt.org/page/UbuntuKVMWalkthrough)) or ((http://www.cyberciti.biz/faq/how-to-install-kvm-on-ubuntu-linux-14-04/)), this is just a collection of some notes which I collected during the evaluation.

General Documentation

RedHat has some good KVM virtualisation documentation available at: https://access.redhat.com/documentation/en/red-hat-enterprise-linux/?version=7/
Debian also has some nice documentation: https://wiki.debian.org/KVM

Creating reproducable VMs for KVM


The easierst way is to use grml-debootstrap to create a new virtual maschine. We already have working netboot environment so we already have added some tuning to grml-debootstrap which makes it even easier.


sudo grml-debootstrap --hostname myfirstvm --vm --target /dev/mapper/vg0-myfirstvm
sudo virt-install --virt-type kvm --name=myfirstmv --vcpu=4 --ram=8192 \
--disk path=/dev/vg0/myfirstvm \
--os-variant=debianwheezy --cpuset=auto --network bridge=br0 --boot hd --vnc

virt-install has many more options(( http://www.techotopia.com/index.php/Installing_a_KVM_Guest_OS_from_the_Command-line_(virt-install) ))
You still need to setup the /etc/network/interfaces file inside your VM!

Examine VM configuration

virsh dumpxml client1

Serial Console for VM

You can use ‘virsh console clientvm’ to connect to the serial console of the virtual machine. In order to make use of it it, you need to activate the serial console in the VM((http://www.cyberciti.biz/faq/howto-setup-serial-console-on-debian-linux/)):

in order to see startup/shtudwon messages:

#/etc/default/grub
GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,19200n8'
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1"

in order to be able to login: (for Debian Jessie with systemd)

systemctl start serial-getty@ttyS0.service
systemctl enable serial-getty@ttyS0.service

Mounting the virutal maschine disk

There are two options available: either with guestfish (install libguestfs-tools (( https://www.async.fi/2016/02/building-virtual-machines-with-vmbuilder/
)) or with virsh virt-edit (install libguestfs-tools ((https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#Accessing_data_on_guest_disk_images))


guestfish --rw --add /dev/vg0/myfistvm


virt-edit NameOfGuest /boot/grub/grub.conf

VM Remote Access with VNC

You need to specify a password in order to make it work with MacOSX buildin VNC client, otherwise the client won’t connect! You can add the password by editing the configuration (( http://www.cyberciti.biz/faq/linux-kvm-vnc-for-guest-machine/ ))

ย <graphics type='vnc' port='-1' autoport='yes' passwd='mysuperduperpassword'/>

Support VM Shutdown

In order to support restart/shutdown from outside the KVM client, you need to install the following packages((http://serverfault.com/questions/549766/kvm-guest-with-acpi-installed-will-not-shutdown)):

– acpid
– acpi-support-base

Change VM parameters

Most parameters (e.g. RAM, CPU) cannot be changed during runtime. You can configure a maximal amount and a (lower) current amount. During runtime of a VM you can only allocate until the defined maximal amount. To increase the maximal value you need to shutdown the VM and change the configuration (( http://serverfault.com/questions/403561/change-amount-of-ram-and-cpu-cores-in-kvm#403671 )).

Lessions learned: CPU Placement

We have one VM which requires a lot of CPU usage (more cores than on one physical CPU). By default KVM seems to to limit one VM to one physical CPU, we need to adjust the settings directly in the XML to use all cpus(( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Manipulating_the_domain_xml-CPU_allocation.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Manipulating_the_domain_xml-CPU_tuning.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Managing_guest_virtual_machines_with_virsh-NUMA_node_management.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Overcommitting_with_KVM-Overcommitting_virtualized_CPUs.html )) (( https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/ch33s08.html )):

before:
<vcpu placement='static' cpuset='0-7'>16</vcpu>
after:
<vcpu placement='static' cpuset='0-15'>16</vcpu>

Summary

There are still many things to discover, rethink and consider for moving from OpenVZ to KVM, e.g. better resources planing as resources cannot be changed as easily as in OpenVZ. So there might be some updates to this post in the future. Stay tuned!

Running “backticks” commands on remote servers

Sometimes it’s necessary to run a complex command on a remote server witch also includes some “backticks”. Usually these commands are interpreted by the local shell so you need to use a little trick to force execution on the remote server:

ssh this.is.my.beautiful.server '( echo `echo "This Command is run on the remote server" ` )'

You need to use single quotes combined with brackets to use the backtick on the remote server.

Cloudflare and Haproxy Lodbalancer

We are currently trying out the cloudflare service to protect one of our company service. In front of this service we are using haproxy as SSL endpoint and loadbalancer. Cloudflare adds a number of custom headers((http://www.linuxorz.com/2014/10/cloudflare-haproxy-get-real-ip/)):

 _SERVER["HTTP_CF_IPCOUNTRY"]      CN
 _SERVER["HTTP_CF_RAY"]            17da8155355b0520-SEA
 _SERVER["HTTP_CF_VISITOR"]        {"scheme":"http"}
 _SERVER["HTTP_CF_CONNECTING_IP"]  XX.YY.ZZ.00

In order to extract the original client IP in the X_FORWARDD_FOR header, you need to use the following configuration((http://permalink.gmane.org/gmane.comp.web.haproxy/12019)) in haproxy:

  acl  FROM_CLOUDFLARE src -f /etc/haproxy/cf-ips-v4
  reqidel  ^X-Forwarded-For:.* if ! LOCALHOST
  reqirep  ^CF-Connecting-IP:(.*)$ X-Forwarded-For:\1 if FROM_CLOUDFLARE
  option  forwardfor if-none

Additionally you need to have the cloudlare IPs in the file /etc/haproxy/cf-ips-v4. You can retrieve their IP ranges from: https://www.cloudflare.com/ips/

Some additional links:

  • https://martensson.io/cloudflare-universal-ssl-with-haproxy/
  • https://github.com/analytically/haproxy-ddos
  • http://fsfe.soup.io/post/244661656/Alexandre-De-Dommelin-weblog-CloudFlare-HAProxy-and

Java SSL Certificate Verification Error

If you come across the situation, that your java programs are not able to connect to ssl encrypted services, it might be most likely that the java cacerts keystore is empty or not uptodate. This might also be due to a bug in the java (or ca-certificate-java) package ((https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1396760)). In order to fix the issue, you can run:

sudo /var/lib/dpkg/info/ca-certificates-java.postinst configure

Hardening SSL

Update: 2014/01/17: Again a few weeks have past without finishing the article. So I’m going to publish it anyway even it it is unfinished work yet. I also disabled OCSP Stapling again. I’m using StartSSL and I’ve had some issues with their OCSP website. Also the nginx implementation is still not “mature” (see: http://nginx.org/patches/attic/ocsp-stapling/README.txt). There are some limitations for less used sites e.g. the OCSP stapling information is stored for each worker. I was getting the “OCSP stapling information outdated” site in my browser often when open my secure site.

Update: 2013/12/21: During writing this post and with the discussion with my colleagues in the last weeks (yeah, it’s already that long since I started writing on this article), the following site came to our attention: Applied Crypto Hardening (https://bettercrypto.org/). This whitepaper is still a draft but already contains a lot more information than this blog post could ever provide ๐Ÿ™‚

It’s the year 2013. An important year concerning security/privacy in the Internet. Because of recent articles in the press I wanted to do a check about the strength of the encryption – both on my private server and also on the websites we are maintaining at the company I work for. And it was a good thing I did this because it turned out that we didn’t update the SSL for year. Especially the configuration on nginx wasn’t “uptodate”. Apparently when we setup the first nginx server we didn’t pay so much attention to that fact and didn’t use far from optiomal settings. Although the settings on Apache were not that suboptimal, they were still 2 or 3 years old and have been copied over and over again without paying attention to them. Anyway after reading some articles (most of the important ones are listed as references at the end of the port).

Forward Secrecy

Forward Secrecy โ€“ also called Perfect Forward Secrecy is a small, but important change in the way the key for the symmetric encryption is exchanged between client and server. In traditional SSL the client sends the session’s symmetric key encrypted with the public key of the server. Someone in the possession of the private key of the server can encrypt the whole communication (this also works for communications happened in the past). Forward Secrecy uses the mathematical principle of Diffie-Hellman to establish a session key by exchanging several messages between client and server and computing the session key out of this messages. Thus the session is key is never sent during the whole process and after the sessions ends and client and server delete the session key the session cannot be decrypted anymore. This also has some drawbacks e.g. higher CPU utilization and slower responses, but security comes with a prices – always ๐Ÿ˜‰ See [1] and [2]

OCSP Stapling

If you are on a recent version of apache or nginx, you can also enable OCSP Stapling. This enhances the former CRL (Certificate Revocation List) und OCSP (Online Certificate Status Protocol). Both of these checks are implemented by the client and the clients needs to verify the certificate at the CA (Certification Authority). With OCSP Stapling the server itself contacts the CA and receive the verification (which is valid for some hours and thus many requests) and sends it back to the client (see [3]).

Strict Transport Security (STS)

The Strict Transport Security (STS) Headers forces a client to only use HTTPS with a site and can be sent on a HTTP request and is cached for a certain amount amount of time, so there are no unencrpyted request after the first initial request.

Removing Compatibility

The most current SSL standard is TLS 1.2. Only the most recent version of servers and browsers are already supporting this standard. On the other hand, SSLv2 is only used by browsers several years old and should be removed in any case. If you need to support WindowsXP and IE, you need to stick to SSLv3, otherwise you should only support TLSv1+. So especially for internal sites where you do not expect such old clients, you should be safely able to remove SSLv3.

Getting a new Certificate

If you are thinking about getting a new certificate you should make sure that your private key uses 4096 bits. Otherwise you will not receive 100 percent with the SSLlabs tests. But still the usual 80% should be pretty fine.

Deciding about the used Cipers

This is a bit complicated than to choose the available protocols. Because you want to force the client to use the best available protocol and you want to support as many clients as possible.
This is basically the same for nginx and Apache: Both use the underlying openssl for the encryption and in both configurations you set an openssl cipher string, which is validated and used by openssl.

After reading a lot of blog entries I decided to go with this ciphers suite:
EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4
And if this is a site which should support WindowsXP + IE you can use this one (which includes RC4):
EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS +RC4 RC4

If you would like to get 100% on the SSLlabs tests you need to exclude some ciphers with “!CAMELLIA128:!ECDSA:AES256-SHA !SEED” (append at the end of list, but before the last RC4).

You can test the cipher with:
$ openssl ciphers -v

nginx

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 5m;
keepalive_timeout 120; # 120 second keep alive

Some generic settings for the HTTP keepalive and the SSL session cache.

ssl_prefer_server_ciphers on;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers $CIPHERS
# Enable this if your want HSTS (recommended, but be careful)
add_header Strict-Transport-Security max-age=15768000 ; includeSubdomains

The first parameters forces the client to use the servers choise of ciphers, the second parameter defines the used protocols (see above), the third options sets the used ciphers (see above).

SSL Compression should be disabled by default with a recent version of nginx and openssl (for details see [5])

OCSP Stapling

If you are using version 1.4, you can enable OCSP Stapling with the following directives (see [4]):


## OCSP Stapling ---
## fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
## the server itself should valid the OCSP before sending to the client
ssl_stapling_verify on;
## verify chain of trust of OCSP response using Root CA and Intermediate certs
ssl_trusted_certificate /path/to/root_CA_cert_plus_intermediates;
## you need to set a rsolver to resolve the OCSP URL
resolver 8.8.8.8;

apache

SSLHonorCipherOrder On
SSLProtocol ALL -SSLv2
SSLCipherSuite $CIPHER

Does basically the same thing as on apache: First force the client to use the best cipher available, define which protocols to use (see above) and at last define which ciphers to use (see above).
You might want to try to disable SSL Compression with (again this depends on the apache version, seee [4] for details):
SSLCompression Off

OCSP Stapling

exim

dovecot

Verifying your changes

One of the best online tests should be the this one: https://www.ssllabs.com/ssltest/

openssl s_client -connect imap.1und1.de:993
openssl s_client -cipher ‘ECDH:DH’ -connect login.live.com:443

openssl ciphers -V

Conclusion

I strongly recommend checking the SSL settings for your site (e.g. via the ssltest website) and changing the configuration accordingly! The final result should look something like this:

ssl-report-arwen-pertl-org

References

  1. Heise Forward Secrecy (in German) http://www.heise.de/security/artikel/Forward-Secrecy-testen-und-einrichten-1932806.html
  2. Hiese Forward Secrecy (in German) http://www.heise.de/security/artikel/Zukunftssicher-Verschluesseln-mit-Perfect-Forward-Secrecy-1923800.html
  3. OCSP Stapling Article at Golem (in German): http://www.golem.de/news/firefox-mitgelieferte-gueltigkeitspruefung-fuer-zertifikate-1307-100700.html
  4. SSLtest by Qualys: https://www.ssllabs.com/ssltest/
  5. Blog Entry by Hynek Schlawack https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  6. OCSP Stapling Blog Article: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  7. Serverdensity Blog Entry about securing Webapps: https://blog.serverdensity.com/how-to-secure-your-webapp/
  8. Lognormal Blog Entry about nginx+ssl: http://www.lognormal.com/blog/2013/06/22/setting-up-ssl-on-nginx/
  9. Blog Entry by Julien Vehent: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  10. Blog Entry by Ivan Ristiฤ‡ http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
  11. Blog Entry by Mike Kuketz http://www.kuketz-blog.de/nsa-abhoersichere-ssl-verschluesselung-fuer-apache-und-nginx/
  12. SSL Rating Guide https://www.ssllabs.com/downloads/SSL_Server_Rating_Guide_2009d.pdf