sysadmin

18 posts

Cloudflare and Haproxy Lodbalancer

We are currently trying out the cloudflare service to protect one of our company service. In front of this service we are using haproxy as SSL endpoint and loadbalancer. Cloudflare adds a number of custom headers((http://www.linuxorz.com/2014/10/cloudflare-haproxy-get-real-ip/)):

 _SERVER["HTTP_CF_IPCOUNTRY"]      CN
 _SERVER["HTTP_CF_RAY"]            17da8155355b0520-SEA
 _SERVER["HTTP_CF_VISITOR"]        {"scheme":"http"}
 _SERVER["HTTP_CF_CONNECTING_IP"]  XX.YY.ZZ.00

In order to extract the original client IP in the X_FORWARDD_FOR header, you need to use the following configuration((http://permalink.gmane.org/gmane.comp.web.haproxy/12019)) in haproxy:

  acl  FROM_CLOUDFLARE src -f /etc/haproxy/cf-ips-v4
  reqidel  ^X-Forwarded-For:.* if ! LOCALHOST
  reqirep  ^CF-Connecting-IP:(.*)$ X-Forwarded-For:\1 if FROM_CLOUDFLARE
  option  forwardfor if-none

Additionally you need to have the cloudlare IPs in the file /etc/haproxy/cf-ips-v4. You can retrieve their IP ranges from: https://www.cloudflare.com/ips/

Some additional links:

  • https://martensson.io/cloudflare-universal-ssl-with-haproxy/
  • https://github.com/analytically/haproxy-ddos
  • http://fsfe.soup.io/post/244661656/Alexandre-De-Dommelin-weblog-CloudFlare-HAProxy-and

Java SSL Certificate Verification Error

If you come across the situation, that your java programs are not able to connect to ssl encrypted services, it might be most likely that the java cacerts keystore is empty or not uptodate. This might also be due to a bug in the java (or ca-certificate-java) package ((https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1396760)). In order to fix the issue, you can run:

sudo /var/lib/dpkg/info/ca-certificates-java.postinst configure

A checklist for creating a “mailout” server with DKIM and SPF

Create the DKIM DNS Record:
http://www.dnswatch.info/dkim/create-dns-record

Create the SPF Record:
http://www.spfwizard.net/

Do not forget to add a PTR record at your provider!

Verify the settings:

Authentication Checker

Use this command:

mail -r xyz@domainthatshouldbeverified.com -s 'testerl vom roman' check-auth-myname=domain.at@verifier.port25.com

And some more information if your domain is hosting email at Google:
https://support.google.com/a/answer/178723?hl=en
https://support.google.com/a/answer/174124?hl=en

Update 2018-01-25: Update port25 authentication checker URL

Hardening SSL

Update: 2014/01/17: Again a few weeks have past without finishing the article. So I’m going to publish it anyway even it it is unfinished work yet. I also disabled OCSP Stapling again. I’m using StartSSL and I’ve had some issues with their OCSP website. Also the nginx implementation is still not “mature” (see: http://nginx.org/patches/attic/ocsp-stapling/README.txt). There are some limitations for less used sites e.g. the OCSP stapling information is stored for each worker. I was getting the “OCSP stapling information outdated” site in my browser often when open my secure site.

Update: 2013/12/21: During writing this post and with the discussion with my colleagues in the last weeks (yeah, it’s already that long since I started writing on this article), the following site came to our attention: Applied Crypto Hardening (https://bettercrypto.org/). This whitepaper is still a draft but already contains a lot more information than this blog post could ever provide πŸ™‚

It’s the year 2013. An important year concerning security/privacy in the Internet. Because of recent articles in the press I wanted to do a check about the strength of the encryption – both on my private server and also on the websites we are maintaining at the company I work for. And it was a good thing I did this because it turned out that we didn’t update the SSL for year. Especially the configuration on nginx wasn’t “uptodate”. Apparently when we setup the first nginx server we didn’t pay so much attention to that fact and didn’t use far from optiomal settings. Although the settings on Apache were not that suboptimal, they were still 2 or 3 years old and have been copied over and over again without paying attention to them. Anyway after reading some articles (most of the important ones are listed as references at the end of the port).

Forward Secrecy

Forward Secrecy – also called Perfect Forward Secrecy is a small, but important change in the way the key for the symmetric encryption is exchanged between client and server. In traditional SSL the client sends the session’s symmetric key encrypted with the public key of the server. Someone in the possession of the private key of the server can encrypt the whole communication (this also works for communications happened in the past). Forward Secrecy uses the mathematical principle of Diffie-Hellman to establish a session key by exchanging several messages between client and server and computing the session key out of this messages. Thus the session is key is never sent during the whole process and after the sessions ends and client and server delete the session key the session cannot be decrypted anymore. This also has some drawbacks e.g. higher CPU utilization and slower responses, but security comes with a prices – always πŸ˜‰ See [1] and [2]

OCSP Stapling

If you are on a recent version of apache or nginx, you can also enable OCSP Stapling. This enhances the former CRL (Certificate Revocation List) und OCSP (Online Certificate Status Protocol). Both of these checks are implemented by the client and the clients needs to verify the certificate at the CA (Certification Authority). With OCSP Stapling the server itself contacts the CA and receive the verification (which is valid for some hours and thus many requests) and sends it back to the client (see [3]).

Strict Transport Security (STS)

The Strict Transport Security (STS) Headers forces a client to only use HTTPS with a site and can be sent on a HTTP request and is cached for a certain amount amount of time, so there are no unencrpyted request after the first initial request.

Removing Compatibility

The most current SSL standard is TLS 1.2. Only the most recent version of servers and browsers are already supporting this standard. On the other hand, SSLv2 is only used by browsers several years old and should be removed in any case. If you need to support WindowsXP and IE, you need to stick to SSLv3, otherwise you should only support TLSv1+. So especially for internal sites where you do not expect such old clients, you should be safely able to remove SSLv3.

Getting a new Certificate

If you are thinking about getting a new certificate you should make sure that your private key uses 4096 bits. Otherwise you will not receive 100 percent with the SSLlabs tests. But still the usual 80% should be pretty fine.

Deciding about the used Cipers

This is a bit complicated than to choose the available protocols. Because you want to force the client to use the best available protocol and you want to support as many clients as possible.
This is basically the same for nginx and Apache: Both use the underlying openssl for the encryption and in both configurations you set an openssl cipher string, which is validated and used by openssl.

After reading a lot of blog entries I decided to go with this ciphers suite:
EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4
And if this is a site which should support WindowsXP + IE you can use this one (which includes RC4):
EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS +RC4 RC4

If you would like to get 100% on the SSLlabs tests you need to exclude some ciphers with “!CAMELLIA128:!ECDSA:AES256-SHA !SEED” (append at the end of list, but before the last RC4).

You can test the cipher with:
$ openssl ciphers -v

nginx

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 5m;
keepalive_timeout 120; # 120 second keep alive

Some generic settings for the HTTP keepalive and the SSL session cache.

ssl_prefer_server_ciphers on;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers $CIPHERS
# Enable this if your want HSTS (recommended, but be careful)
add_header Strict-Transport-Security max-age=15768000 ; includeSubdomains

The first parameters forces the client to use the servers choise of ciphers, the second parameter defines the used protocols (see above), the third options sets the used ciphers (see above).

SSL Compression should be disabled by default with a recent version of nginx and openssl (for details see [5])

OCSP Stapling

If you are using version 1.4, you can enable OCSP Stapling with the following directives (see [4]):


## OCSP Stapling ---
## fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
## the server itself should valid the OCSP before sending to the client
ssl_stapling_verify on;
## verify chain of trust of OCSP response using Root CA and Intermediate certs
ssl_trusted_certificate /path/to/root_CA_cert_plus_intermediates;
## you need to set a rsolver to resolve the OCSP URL
resolver 8.8.8.8;

apache

SSLHonorCipherOrder On
SSLProtocol ALL -SSLv2
SSLCipherSuite $CIPHER

Does basically the same thing as on apache: First force the client to use the best cipher available, define which protocols to use (see above) and at last define which ciphers to use (see above).
You might want to try to disable SSL Compression with (again this depends on the apache version, seee [4] for details):
SSLCompression Off

OCSP Stapling

exim

dovecot

Verifying your changes

One of the best online tests should be the this one: https://www.ssllabs.com/ssltest/

openssl s_client -connect imap.1und1.de:993
openssl s_client -cipher ‘ECDH:DH’ -connect login.live.com:443

openssl ciphers -V

Conclusion

I strongly recommend checking the SSL settings for your site (e.g. via the ssltest website) and changing the configuration accordingly! The final result should look something like this:

ssl-report-arwen-pertl-org

References

  1. Heise Forward Secrecy (in German) http://www.heise.de/security/artikel/Forward-Secrecy-testen-und-einrichten-1932806.html
  2. Hiese Forward Secrecy (in German) http://www.heise.de/security/artikel/Zukunftssicher-Verschluesseln-mit-Perfect-Forward-Secrecy-1923800.html
  3. OCSP Stapling Article at Golem (in German): http://www.golem.de/news/firefox-mitgelieferte-gueltigkeitspruefung-fuer-zertifikate-1307-100700.html
  4. SSLtest by Qualys: https://www.ssllabs.com/ssltest/
  5. Blog Entry by Hynek Schlawack https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  6. OCSP Stapling Blog Article: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  7. Serverdensity Blog Entry about securing Webapps: https://blog.serverdensity.com/how-to-secure-your-webapp/
  8. Lognormal Blog Entry about nginx+ssl: http://www.lognormal.com/blog/2013/06/22/setting-up-ssl-on-nginx/
  9. Blog Entry by Julien Vehent: https://jve.linuxwall.info/blog/index.php?post/2013/10/12/A-grade-SSL/TLS-with-Nginx-and-StartSSL
  10. Blog Entry by Ivan Ristić http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html
  11. Blog Entry by Mike Kuketz http://www.kuketz-blog.de/nsa-abhoersichere-ssl-verschluesselung-fuer-apache-und-nginx/
  12. SSL Rating Guide https://www.ssllabs.com/downloads/SSL_Server_Rating_Guide_2009d.pdf

Syslog logging with Cisco ASA

In the last week I was tweaking the logging setup of our Cisco ASA firewalls at work and find out why it didn’t work in the first place and how to disable “unneeded” messages. Again this post is nothing you won’t somewhere on the Internet or in the Cisco Documentation or by cafefully looking the ASDM interface.

First you need to setup to which server you are want to log. The settings should be pretty straight forward. You cannot use the standard port TCP 514 with Cisco ASA, so we setup d a DNAT on the syslog server from port TCP 1470 to TCP 514. The commandline option is:
logging host interface-name syslog-ip-address proto/port

Cisco-ASA-Syslog-Syslog-Servers

Cisco-ASA-Syslog-TCP-Port

There’s an important option at the top of the page. The option allows traffic to in case the syslog servers is down (only works with TCP syslog of course). I don’t find the idea of introduction a dependency between the syslog server and the firewall a good idea (at least if you use graylog which wasn’t very stable in the past, although it has improved in the latest versions). The commandline option is:
logging permit-hostdown

Cisco-ASA-Allow-Traffic-When-Syslog-Down

The Cisco ASA doesn’t send the hostname by default (tested on version 8.4). In order to get the Cisco ASA to send the hostname in the syslog message you need to enable the following command
logging device-id hostname
I don’t know where to find this option in the ASDM.

It’s not enough to configre the syslog server to get it working. You also need to enable it in the syslog filter and setup which syslog levels you want to log to syslog, via email etc. You can define custom map of filters based on event class and severity or just filter on serverty. I find the level informative to be the best one if you disable some messages which produce a lot of messages in the next step. It is crucial to have syslog not disabled on this page, otherwise there will be no logging to your syslog server.

Cisco-ASA-Syslog-Filters

The commandline options are:
logging trap informational
logging asdm informational

The last step is do define which logging messages the ASA should log which which serverty, e.g. you can define that “syslog id” e.g. 105005:
%ASA-1-105005: (Primary) Lost Failover communications with mate on interface interface_name.

Cisco-ASA-Syslog-Disable-Conntrack-Messages

I have found that the connection tracking is very “informative” and logs each connection creation and teardown despite if you enable or disable logging of the firewall rule. So I disabled these “syslog ids” in order to have a readable logfiles. This setting also applies to the logging window you can open in the ASDM.

You can also disable those “syslog id” in the commandline:
no logging message 305012
no logging message 305011
no logging message 305012
no logging message 302012
no logging message 302013
no logging message 302014
no logging message 302015
no logging message 302016
no logging message 302020
no logging message 302021

The option “log timestamps” sounds good, but we had problems with this option on our central syslog server server (graylog2). After enabling the option graylog could not correctly parse the syslog message and wouldn’t log the message with the correct hostname.

So now we have a working syslog setup wich our ASAs which only contain the syslog message we would like to have.

Learning something new every day…

One of the things I really like about working within a team in one location is the fact that are able to learn something new regularly. And of course you are also able to give some useful advice to others πŸ™‚

Something I learned this week is how to turn on the UID light on a HP server without logging into the iLO webinterface. You can use the tool hpasmcli – which probably can do some more useful stuff too πŸ™‚


# hpasmcli
HP management CLI for Linux (v2.0)
Copyright 2008 Hewlett-Packard Development Group, L.P.

--------------------------------------------------------------------------
NOTE: Some hpasmcli commands may not be supported on all Proliant servers.
Type 'help' to get a list of all top level commands.
--------------------------------------------------------------------------
hpasmcli> set uid on
UID is now on.
hpasmcli> set uid off
UID is now off.
hpasmcli> quit

“I solved it” – Java with MacOSX

Many of you have probably noticed during the last months. Java is somewhat broken with MacOSX since Oracle started shipping Java7 for MacOSX. Some of the things I recommend about using Java on MacOSX:

– Install the SDK even if you do not necessary need it as it might fix some problems
– If you upgrade Java7 and you need Java6 for some applications (e.g. in my case it was Cisco ASDM configuration for Cisco ASA), you might need to reinstall Java6 (latest version as of July 2013 is here: https://support.apple.com/kb/dl1572 ). I have no idea what it does, but it does something with the binaries in /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/ – sizes seem to be the same, I don’t have any md5sums…

Hopefully that should fix most of the Java issues.