Update for the Checklist on “mailout” servers

This is an update to the checklist to create a prefect mailout server:
Original Checklist

Setup DMARC DNS Record to receive mail delivery reports

btw: I Just started adding all those settings to my own domain too. Google DKIM signing is still waiting for DNS propagation.

Cloudflare and Haproxy Lodbalancer

We are currently trying out the cloudflare service to protect one of our company service. In front of this service we are using haproxy as SSL endpoint and loadbalancer. Cloudflare adds a number of custom headers1)http://www.linuxorz.com/2014/10/cloudflare-haproxy-get-real-ip/ :

 _SERVER["HTTP_CF_RAY"]            17da8155355b0520-SEA
 _SERVER["HTTP_CF_VISITOR"]        {"scheme":"http"}

In order to extract the original client IP in the X_FORWARDD_FOR header, you need to use the following configuration2)http://permalink.gmane.org/gmane.comp.web.haproxy/12019 in haproxy:

  acl  FROM_CLOUDFLARE src -f /etc/haproxy/cf-ips-v4
  reqidel  ^X-Forwarded-For:.* if ! LOCALHOST
  reqirep  ^CF-Connecting-IP:(.*)$ X-Forwarded-For:\1 if FROM_CLOUDFLARE
  option  forwardfor if-none

Additionally you need to have the cloudlare IPs in the file /etc/haproxy/cf-ips-v4. You can retrieve their IP ranges from: https://www.cloudflare.com/ips/

Some additional links:

  • https://martensson.io/cloudflare-universal-ssl-with-haproxy/
  • https://github.com/analytically/haproxy-ddos
  • http://fsfe.soup.io/post/244661656/Alexandre-De-Dommelin-weblog-CloudFlare-HAProxy-and

References   [ + ]

1. http://www.linuxorz.com/2014/10/cloudflare-haproxy-get-real-ip/
2. http://permalink.gmane.org/gmane.comp.web.haproxy/12019

Java SSL Certificate Verification Error

If you come across the situation, that your java programs are not able to connect to ssl encrypted services, it might be most likely that the java cacerts keystore is empty or not uptodate. This might also be due to a bug in the java (or ca-certificate-java) package 1)https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1396760 . In order to fix the issue, you can run:

sudo /var/lib/dpkg/info/ca-certificates-java.postinst configure

References   [ + ]

1. https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1396760

A checklist for creating a “mailout” server with DKIM and SPF

Create the DKIM DNS Record:

Create the SPF Record:

Do not forget to add a PTR record at your provider!

Verify the settings:

Authentication Checker

Use this command:

mail -r xyz@domainthatshouldbeverified.com -s 'testerl vom roman' check-auth-myname=domain.at@verifier.port25.com

And some more information if your domain is hosting email at Google:

Update 2018-01-25: Update port25 authentication checker URL

Hardening SSL

Update: 2014/01/17: Again a few weeks have past without finishing the article. So I’m going to publish it anyway even it it is unfinished work yet. I also disabled OCSP Stapling again. I’m using StartSSL and I’ve had some issues with their OCSP website. Also the nginx implementation is still not “mature” (see: http://nginx.org/patches/attic/ocsp-stapling/README.txt). There are some limitations for less used sites e.g. the OCSP stapling information is stored for each worker. I was getting the “OCSP stapling information outdated” site in my browser often when open my secure site.

Update: 2013/12/21: During writing this post and with the discussion with my colleagues in the last weeks (yeah, it’s already that long since I started writing on this article), the following site came to our attention: Applied Crypto Hardening (https://bettercrypto.org/). This whitepaper is still a draft but already contains a lot more information than this blog post could ever provide 🙂

It’s the year 2013. An important year concerning security/privacy in the Internet. Because of recent articles in the press I wanted to do a check about the strength of the encryption – both on my private server and also on the websites we are maintaining at the company I work for. And it was a good thing I did this because it turned out that we didn’t update the SSL for year. Especially the configuration on nginx wasn’t “uptodate”. Apparently when we setup the first nginx server we didn’t pay so much attention to that fact and didn’t use far from optiomal settings. Although the settings on Apache were not that suboptimal, they were still 2 or 3 years old and have been copied over and over again without paying attention to them. Anyway after reading some articles (most of the important ones are listed as references at the end of the port).

Forward Secrecy

Forward Secrecy – also called Perfect Forward Secrecy is a small, but important change in the way the key for the symmetric encryption is exchanged between client and server. In traditional SSL the client sends the session’s symmetric key encrypted with the public key of the server. Someone in the possession of the private key of the server can encrypt the whole communication (this also works for communications happened in the past). Forward Secrecy uses the mathematical principle of Diffie-Hellman to establish a session key by exchanging several messages between client and server and computing the session key out of this messages. Thus the session is key is never sent during the whole process and after the sessions ends and client and server delete the session key the session cannot be decrypted anymore. This also has some drawbacks e.g. higher CPU utilization and slower responses, but security comes with a prices – always 😉 See [1] and [2]

OCSP Stapling

If you are on a recent version of apache or nginx, you can also enable OCSP Stapling. This enhances the former CRL (Certificate Revocation List) und OCSP (Online Certificate Status Protocol). Both of these checks are implemented by the client and the clients needs to verify the certificate at the CA (Certification Authority). With OCSP Stapling the server itself contacts the CA and receive the verification (which is valid for some hours and thus many requests) and sends it back to the client (see [3]).

Strict Transport Security (STS)

The Strict Transport Security (STS) Headers forces a client to only use HTTPS with a site and can be sent on a HTTP request and is cached for a certain amount amount of time, so there are no unencrpyted request after the first initial request.

Removing Compatibility

The most current SSL standard is TLS 1.2. Only the most recent version of servers and browsers are already supporting this standard. On the other hand, SSLv2 is only used by browsers several years old and should be removed in any case. If you need to support WindowsXP and IE, you need to stick to SSLv3, otherwise you should only support TLSv1+. So especially for internal sites where you do not expect such old clients, you should be safely able to remove SSLv3.

Getting a new Certificate

If you are thinking about getting a new certificate you should make sure that your private key uses 4096 bits. Otherwise you will not receive 100 percent with the SSLlabs tests. But still the usual 80% should be pretty fine.

Deciding about the used Cipers

This is a bit complicated than to choose the available protocols. Because you want to force the client to use the best available protocol and you want to support as many clients as possible.
This is basically the same for nginx and Apache: Both use the underlying openssl for the encryption and in both configurations you set an openssl cipher string, which is validated and used by openssl.

After reading a lot of blog entries I decided to go with this ciphers suite:
And if this is a site which should support WindowsXP + IE you can use this one (which includes RC4):

If you would like to get 100% on the SSLlabs tests you need to exclude some ciphers with “!CAMELLIA128:!ECDSA:AES256-SHA !SEED” (append at the end of list, but before the last RC4).

You can test the cipher with:
$ openssl ciphers -v


ssl_session_cache shared:SSL:50m;
ssl_session_timeout 5m;
keepalive_timeout 120; # 120 second keep alive

Some generic settings for the HTTP keepalive and the SSL session cache.

ssl_prefer_server_ciphers on;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers $CIPHERS
# Enable this if your want HSTS (recommended, but be careful)
add_header Strict-Transport-Security max-age=15768000 ; includeSubdomains

The first parameters forces the client to use the servers choise of ciphers, the second parameter defines the used protocols (see above), the third options sets the used ciphers (see above).

SSL Compression should be disabled by default with a recent version of nginx and openssl (for details see [5])

OCSP Stapling

If you are using version 1.4, you can enable OCSP Stapling with the following directives (see [4]):

## OCSP Stapling ---
## fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
## the server itself should valid the OCSP before sending to the client
ssl_stapling_verify on;
## verify chain of trust of OCSP response using Root CA and Intermediate certs
ssl_trusted_certificate /path/to/root_CA_cert_plus_intermediates;
## you need to set a rsolver to resolve the OCSP URL


SSLHonorCipherOrder On
SSLProtocol ALL -SSLv2
SSLCipherSuite $CIPHER

Does basically the same thing as on apache: First force the client to use the best cipher available, define which protocols to use (see above) and at last define which ciphers to use (see above).
You might want to try to disable SSL Compression with (again this depends on the apache version, seee [4] for details):
SSLCompression Off

OCSP Stapling



Verifying your changes

One of the best online tests should be the this one: https://www.ssllabs.com/ssltest/

openssl s_client -connect imap.1und1.de:993
openssl s_client -cipher ‘ECDH:DH’ -connect login.live.com:443

openssl ciphers -V


I strongly recommend checking the SSL settings for your site (e.g. via the ssltest website) and changing the configuration accordingly! The final result should look something like this:



  1. Heise Forward Secrecy (in German) http://www.heise.de/security/artikel/Forward-Secrecy-testen-und-einrichten-1932806.html
  2. Hiese Forward Secrecy (in German) http://www.heise.de/security/artikel/Zukunftssicher-Verschluesseln-mit-Perfect-Forward-Secrecy-1923800.html
  3. OCSP Stapling Article at Golem (in German): http://www.golem.de/news/firefox-mitgelieferte-gueltigkeitspruefung-fuer-zertifikate-1307-100700.html
  4. SSLtest by Qualys: https://www.ssllabs.com/ssltest/
  5. Blog Entry by Hynek Schlawack https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  6. OCSP Stapling Blog Article: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  7. Serverdensity Blog Entry about securing Webapps: https://blog.serverdensity.com/how-to-secure-your-webapp/
  8. Lognormal Blog Entry about nginx+ssl: http://www.lognormal.com/blog/2013/06/22/setting-up-ssl-on-nginx/
  9. Blog Entry by Julien Vehent: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  10. Blog Entry by Ivan Ristić http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
  11. Blog Entry by Mike Kuketz http://www.kuketz-blog.de/nsa-abhoersichere-ssl-verschluesselung-fuer-apache-und-nginx/
  12. SSL Rating Guide https://www.ssllabs.com/downloads/SSL_Server_Rating_Guide_2009d.pdf

A few Updates: Upgrade to Nginx 1.4, PHP 5.5, Munin 2.0

Today (well, actually it’s already some days ago on the day of the release of this post) I did some updates on the server. The server is now running nginx 1.4 (more about this change in one of the following blog articles), PHP 5.5 and Munin 2.0. All in all everything went pretty smooth, I had to disable one of my WordPress Plugins because it was not compatible with PHP 5.5 but as it was unmaintained for over 2 years and not important I just deleted it. All of the new software is in Personal Package Archives (ppa) repositories, so they are not part of the official Ubuntu Distribution. But all of the repositories (or at least the important ones: nginx and php) look maintained good enough. At least for my personal server.

add-apt-repository ppa:nginx/stable
add-apt-repository ppa:ondrej/php5
add-apt-repository ppa:josh-boon/munin

And then just run a normal update. Your software should be automatically updated! I also installed a new WordPress Plugin named W3 Total Cache with a nice memcache server, so my blog should be faster than ever 🙂

I still need to get some nice munin graphs for the new opCache in PHP 5.5. Remember that some things have changed in PHP 5.5. You might want to remove some config files on your own (e.g. suhosin.ini). suhosin seems to not maintained anymore and thus is not included in the PHP 5.5 packages. I’ve also had to path the upstart script to create /var/run/php5-fpm (for the sockets). Wev’ve encountered a similar problem at work this week with a different distrubution and a different repositories too.

Getting Munin 2.0 up and running with nginx is a bit tricky but there’s already good documentation out there. And just to be sure, here’s the correct nginx configuration:

    location /munin/static/ {
        auth_basic            "restricted";
        auth_basic_user_file  /etc/htpasswd;
        alias /etc/munin/static/;

    location  ^~  /munin-cgi/munin-cgi-graph/ {
        auth_basic            "restricted";
        auth_basic_user_file  /etc/htpasswd;
        fastcgi_split_path_info ^(/munin-cgi/munin-cgi-graph)(.*);
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass unix:/var/run/munin/fastcgi-munin-graph.sock;
        include fastcgi_params;

    location /munin/ {
        auth_basic            "restricted";
        auth_basic_user_file  /etc/htpasswd;
        fastcgi_split_path_info ^(/munin)(.*);
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass unix:/var/run/munin/fastcgi-munin-html.sock;
        include fastcgi_params;
  • Official Documentation about munin+nginx: http://munin.readthedocs.org/en/latest/example/webserver/nginx.html
  • A bit more in detail configuration: http://uname.pingveno.net/blog/index.php/post/2013/08/25/Configure-Munin-graphs-with-Nginx-and-Debian-7 Please note the nginx vhost on this site has the possibility to restrict munin users to a particular group. I recommend sticking with the more simpler configuration from the link above.

Syslog logging with Cisco ASA

In the last week I was tweaking the logging setup of our Cisco ASA firewalls at work and find out why it didn’t work in the first place and how to disable “unneeded” messages. Again this post is nothing you won’t somewhere on the Internet or in the Cisco Documentation or by cafefully looking the ASDM interface.

First you need to setup to which server you are want to log. The settings should be pretty straight forward. You cannot use the standard port TCP 514 with Cisco ASA, so we setup d a DNAT on the syslog server from port TCP 1470 to TCP 514. The commandline option is:
logging host interface-name syslog-ip-address proto/port



There’s an important option at the top of the page. The option allows traffic to in case the syslog servers is down (only works with TCP syslog of course). I don’t find the idea of introduction a dependency between the syslog server and the firewall a good idea (at least if you use graylog which wasn’t very stable in the past, although it has improved in the latest versions). The commandline option is:
logging permit-hostdown


The Cisco ASA doesn’t send the hostname by default (tested on version 8.4). In order to get the Cisco ASA to send the hostname in the syslog message you need to enable the following command
logging device-id hostname
I don’t know where to find this option in the ASDM.

It’s not enough to configre the syslog server to get it working. You also need to enable it in the syslog filter and setup which syslog levels you want to log to syslog, via email etc. You can define custom map of filters based on event class and severity or just filter on serverty. I find the level informative to be the best one if you disable some messages which produce a lot of messages in the next step. It is crucial to have syslog not disabled on this page, otherwise there will be no logging to your syslog server.


The commandline options are:
logging trap informational
logging asdm informational

The last step is do define which logging messages the ASA should log which which serverty, e.g. you can define that “syslog id” e.g. 105005:
%ASA-1-105005: (Primary) Lost Failover communications with mate on interface interface_name.


I have found that the connection tracking is very “informative” and logs each connection creation and teardown despite if you enable or disable logging of the firewall rule. So I disabled these “syslog ids” in order to have a readable logfiles. This setting also applies to the logging window you can open in the ASDM.

You can also disable those “syslog id” in the commandline:
no logging message 305012
no logging message 305011
no logging message 305012
no logging message 302012
no logging message 302013
no logging message 302014
no logging message 302015
no logging message 302016
no logging message 302020
no logging message 302021

The option “log timestamps” sounds good, but we had problems with this option on our central syslog server server (graylog2). After enabling the option graylog could not correctly parse the syslog message and wouldn’t log the message with the correct hostname.

So now we have a working syslog setup wich our ASAs which only contain the syslog message we would like to have.

Konzertbericht – Circle of Illusion – Jeremia

Nachdem ich es mir jetzt schon so oft vorgenommen haben, einen Bericht ĂŒber ein besuchtes Konzert zu schreiben, hier nun mein erster Konzertbericht. Mittlerweile ist das Konzert ja auch schon wieder ĂŒber eine Woche her…
Eigentlich bin ich nur durch Zufall beim DurchblĂ€ttern des Planet.tt Magazins auf die Band aufmerksam geworden. Die Beschreibung klang nach einem sehr vielversprechenden Konzert. Das Konzert war zugleich auch eine Art offizielle CD-PrĂ€sentations-Event. Dementsprechend war das Publikum bunt gemischt und dem Applaus fĂŒr die einzelnen Bandmitglieder zu urteilen, waren wohl viele Verwandte/Freunde/Bekannte der Bandmitglieder anwesend. Beim Betreten der Szene Wien bot sich dann ein ungewohntes Bild, denn der komplette Saal war bestuhlt. Trotzdem bot der Abend ein epochales Rack-Opern-Musical ausgefĂŒhrt mit vielen Duetten zwischen den drei SĂ€ngern, untermalt von griffigem Rock/Metal-Musik, unterstĂŒtzt von einer Violinistin und einzelnen “Funk”/Elekro-Elementen aus dem Keyboard. Das gesamte Konzert/Ablum folgt einer duchgĂ€ngigen Story (die sowohl auf den PlĂ€tzen im Saal als auch im Booklet der CD nachgelesen werden kann), die sich um die Charaktere Jeremy, Jelena und Sarah dreht. In einem Satz zusammengefasst kann sich der Inhalt so zusammenfasst werden: Jeremy ist in der Traumwelt von Jelena eingesperrt und muss seine Fehlverhalten gegenĂŒber seiner Freundin Sarah erkennen um daraus auszubrechen. Detailiertere Information, insbesondere zur Zusammensetzung der Band gibt es natĂŒrlich auf deren Homepage: http://www.circleofillusion.com/
Auch die CD wurde schon mehrmals angehört. Sicherlich kein Album, dass man sich jeden Tag anhört, dafĂŒr fehlen vor allem ein wenig prĂ€gnante Refrains, die einem im GedĂ€chtnis bleiben. Dennoch ist die CD sehr empfehlenswert und sowohl CD als auch Konzert bekommt 8 von 10 Sternen auf meiner persönlichen Skala.


Learning something new every day…

One of the things I really like about working within a team in one location is the fact that are able to learn something new regularly. And of course you are also able to give some useful advice to others 🙂

Something I learned this week is how to turn on the UID light on a HP server without logging into the iLO webinterface. You can use the tool hpasmcli – which probably can do some more useful stuff too 🙂

# hpasmcli
HP management CLI for Linux (v2.0)
Copyright 2008 Hewlett-Packard Development Group, L.P.

NOTE: Some hpasmcli commands may not be supported on all Proliant servers.
Type 'help' to get a list of all top level commands.
hpasmcli> set uid on
UID is now on.
hpasmcli> set uid off
UID is now off.
hpasmcli> quit