haproxy as a very very overloaded sslh

After using haproxy at work for some time I realized that it can be configured for a lot of things, for example: it knows about SNI (on ssl is the method we use to know what host the client is trying to reach so that we know what certificate to present and thus we can multiplex several virtual hosts on the same ssl IP:port) and it also knows how to make transparent proxy connections (the connections go through haproxy but the ending server will think they are arriving directly from the client, as it will see the client's IP as the source IP of the packages).

With this two little features, which are available on haproxy 1.5 (Jessie's version has them all), I thought I could give it a try to substitute sslh with haproxy giving me a lot of possibilities that sslh cannot do.

Having this in mind I thought I could multiplex several ssl services, not only https but also openvpn or similar, on the 443 port and also allow this services to arrive transparently to the final server. Thus what I wanted was not to mimic sslh (which can be done with haproxy) but to get the semantic I needed, which is similar to sslh but with more power and with a little different behaviour, cause I liked it that way.

There is however one caveat that I don't like about this setup and it is that to achieve the transparency one has to run haproxy as root, which is not really something one likes :-( so, having transparency is great, but we'll be taking some risks here which I personally don't like, to me it isn't worth it.

Anyway, here is the setup, it basically consists of a setup on haproxy but if we want transparency we'll have to add to it a routing and iptables setup, I'll describe here the whole setup

Here is what you need to define on /etc/haproxy/haproxy.cfg:

frontend ft_ssl bind 192.168.0.1:443 mode tcp option tcplog tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 } acl sslvpn req_ssl_sni -i vpn.example.net use_backend bk_sslvpn if sslvpn use_backend bk_web if { req_ssl_sni -m found } default_backend bk_ssh backend bk_sslvpn mode tcp source 0.0.0.0 usesrc clientip server srvvpn vpnserver:1194 backend bk_web mode tcp source 0.0.0.0 usesrc clientip server srvhttps webserver:443 backend bk_ssh mode tcp source 0.0.0.0 usesrc clientip server srvssh sshserver:22

An example of a transparent setup can be found here but lacks some details, for example, if you need to redirect the traffic to the local haproxy you'll want to use the xt_TPROXY, there is a better doc for that at squid's wiki. Anyway, if you are playing just with your own machine, like we typically do with sslh, you won't need the TPROXY power, as packets will come straight to your 443, so haproxy will be able to get the without any problem. The problem will come if you are using transparency (source 0.0.0.0 usesrc clientip) because then packets coming out of haproxy will be carrying the ip of the real client, and thus the answers of the backend will go to that client (but with different ports and other tcp data), so it will not work. We'll have to get those packets back to haproxy, for that what we'll do is mark the packages with iptables and then route them to the loopback interface using advanced routing. This is where all the examples will tell you to use iptables' mangle table with rules marking on PREROUTING but that won't work out if you are having all the setup (frontend and backends) in just one box, instead you'll have to write those rules to work on the OUTPUT chain of the mangle table, having something like this:

*mangle :PREROUTING ACCEPT :INPUT ACCEPT :FORWARD ACCEPT :OUTPUT ACCEPT :POSTROUTING ACCEPT :DIVERT - -A OUTPUT -s public_ip -p tcp --sport 22 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 443 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 1194 -o public_iface -j DIVERT -A DIVERT -j MARK --set-mark 1 -A DIVERT -j ACCEPT COMMIT

Take that just as an example, better suggestions on how to know what traffic to send to DIVERT are welcome. The point here is that if you are sending the service to some other box you can do it on PREROUTIING, but if you are sending the service to the very same box of haproxy you'll have to mark the packages on the OUTPUT chain.

Once we have the packets marked we just need to route them, something like this will work out perfectly:

ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100

And that's all for this crazy setup. Of course, if, like me, you don't like the root implication of the transparent setup, you can remove the "source 0.0.0.0 usesrc clientip" lines on the backends and forget about transparency (connections to the backend will come from your local IP), but you'll be able to run haproxy with dropped privileges and you'll just need the plain haproxy.cfg setup and not the weird iptables and advanced routing setup.

Hope you like the article, btw, I'd like to point out the main difference of this setup vs sslh, it is that I'm only sending the packages to the ssl providers if the client is sending SNI info, otherwise I'm sending them to the ssh server, while sslh will send ssl clients without SNI also to the ssl provider. If your setup mimics sslh and you want to comment on it, feel free to do it.

Add a Debian source repository to your ubuntu

Debian does all the work that Ubuntu bases its system on, so... if you want to add our latest Debian source package repository to ubuntu, so that you can compile and use it under Ubuntu, that should be as easy as to add to your /etc/apt/sources.list file:

deb-src http://ftp.debian.org/debian/ testing main deb-src http://ftp.debian.org/debian/ unstable main deb-src http://ftp.debian.org/debian/ ../project/experimental main

Well, just the line you want, if you want the future version of Debian (currently Jessie) use the testing one, if you want current develpment, use unstable, and sometimes you even get a experimental version if you want to test really bleeding edge versions.

The problem here is that after you add the lines you want to your sources.list file and you run your

apt-get update

You will end with a GPG error because the ubuntu's apt-key keyring doesn't know about Debian's keys, so... we'll have to run a few commands to get rid of this, but first we must locate the needed key for example here

wget https://ftp-master.debian.org/keys/archive-key-7.0.asc -O -|apt-key add - apt-get update

Now hopefully your Ubuntu will recognize your Debian sources and you'll be able to get your Debian favourite source into your good old Ubuntu by doing something like this as user (we'll use fakeroot):

apt-get install fakeroot apt-get apt-get build-dep your_favourite_package apt-get source your_favourite_package cd your_favourite_package_source_dir dpkg-buildpackage -rfakeroot

Squid proxy being transparent also for ssl and other tcp connections by using ssl bump

A long time ago I was trying to have a transparent proxy setup by using squid, but squid traditionally only knows about http, ftp and https in explicit proxy mode. There is no way to handle non http (for example https) transparently on a traditional setup, so that setup was not what I was looking for.

After looking into TLS SNI and other things, trying even to implement things like that on some tools like socat that would proxify things for squid, I discovered ssl bump on squid3, which just does all the magic I was looking for.

Squid traditionally has several ways of listening to requests, one of which is the explicit proxy port (http_port 3128) and the other typical ones are for transparent proxy, this is indicated by flags:

• transparent: used to intercept server queries and parse http host headers to forward them through squid to the servers
• tproxy: used to spoof outgoing address to that of the client, so that squid is really transparent

Well, none of this allows us to forward https or tcp requests (say for example ssh, imap, ...) for a client that doesn't have explicit proxy support. Unluckily this means that a transparent proxy using this technology nowadays is of no use.

This is where ssl bump comes to the rescue, the old transparent mode of squid, which is currently called intercept, on squid3 has an extra flag: ssl-bump, which has the power of being able to intercept ssl traffic and things like that, allowing squid to cache https webs, but to do this, one has to create a Certificate Authority and the clients must trust this CA that squid uses to issue certificates for the web sites we want to visit.

However ssl-bump can work without issuing these certificates and in this case squid won't mess with https requests, but it will still allow us to do a pretty neat thing, which is to forward all tcp connections from any client (doesn't even have to know what a proxy is) transparently. In this case what squid does is to ask netfilter (iptables) where was the connection that squid is handling supposed to go, and squid makes this connection for the client so that it starts talking to the other end with all the traffic going through squid.

One might ask himself why would he want this traffic on squid, well, you'll have all of squid features, you can control the speed, you get all the typical logs and acls, ...

Of course that if you don't want this you can go with iptables and traffic shaper and that's good as well.

So... you like this idea? Well, then if your distro has squid compiled with ssl support you can read the config section, but if you are (like me) using Debian, you must recompile your squid3 with ssl support. Debian doesn't compile squid3 with ssl support as there are problems between openssl license and squid3 one (squid developers are looking forward to somebody porting the code to gnutls :-)

Rebuilding squid3 with ssl-bump

Well, to rebuild the squid3 package with ssl support you must install the needed packages:

apt-get install fakeroot libssl-dev apt-get build-dep squid3

There you may find that your distro doesn't have all the packages needed to compile, like for example libecap2-dev, in this case you'll have to apt-get source these packages, compile and install them like we'll do with squid3

And then do a few things as user (we'll use fakeroot which we have just installed) I've tested this using squid3 from Debian testing (the next version, which will be Jessie)

Start with:

apt-get source squid3

And then we'll edit a couple of files on the source, so cd into the source dir and in the debian/control file you must add to the build-depends: libssl-dev and in the debian/rules file you must add this configure options:

--enable-ssl \ --enable-ssl-crtd \

One can then run debchange -i and add something like this on the changelog:

Build with --enable-ssl and --enable-ssl-crtd.

Now the source is ready to build using your favourite command, like for example:

dpkg-buildpackage -rfakeroot

and at last install the needed packages using dpkg -i

Configuration

The main configuration file is stored at /etc/squid3/squid.conf, even though it can be split into separate files. On these files we must set as SSL_ports all the ports for the protocols that we want to allow through squid using an acl like this:

acl SSL_ports port 1935 # rtmp acl SSL_ports port 5222 # xmpp acl SSL_ports port 5223 # xmpp over ssl acl SSL_ports port 5228 # googletalk acl SSL_ports port 5242 # viber acl SSL_ports port 4244 # viber

And we'd typically want to define several proxy ports, one for explicit http, another one for http interception (classic transparent proxy) and our ssl-bump port, like this:

http_port 3128 http_port 80 intercept https_port 3127 intercept ssl-bump generate-host-certificates=off cert=/etc/squid3/squid.pem acl ssl-bump_port myportname 3127 always_direct allow ssl-bump_port

The direct thing is to force the squid server to send ssl-bump requests directly and not through other caches, as this wouldn't work at all. The certificate is needed even though we won't be using it, just generate one using make-ssl-cert from the ssl-cert package or plain openssl x509 power.

Make sure we have all our client networks listed on our localnet acl and that they are allowed to use the proxy:

acl localnet src 127.0.0.1/32 192.168.0.0/16 http_access allow localnet

Several misc settings like: make sure that ssl-bump doesn't generate certificates for any host at all (just in case), set the language for the errors and allow a good number of filedescriptors so that we don't run out of them

ssl_bump none all error_default_language es max_filedescriptors 8192

And that is pretty much what is needed on the squid configuration, of course you can do this and more writing it all in different ways. I have found out that the directory for the ssl certs is not created by default, we must run: /usr/lib/squid3/ssl_crtd -c -s /var/lib/ssl_db

Routing it all through squid

You may be wondering how does all the traffic that we are going to allow through squid get to it. Well, the answer is easy if you have ever configured some kind of transparent proxy or similar, we do it through iptables, in the PREROUTING chain of the nat table we send things to the ssl-bump port or to our transparent proxy port as we like and then we open the ports on which we are serving all this on the INPUT chain of the filter table with something like this which can be loaded with iptables-restore myiptables.cfg:

*nat :PREROUTING ACCEPT :POSTROUTING ACCEPT :OUTPUT ACCEPT -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 1935 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 3389 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 5222 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 5223 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 5228 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 5242 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 4244 -j REDIRECT --to-ports 3127 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 80 -j REDIRECT --to-ports 80 #This is not for squid but I like these too (local ntp and dns cache) -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p udp --dport ntp -j REDIRECT --to-ports 123 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport domain -j REDIRECT --to-port 53 -A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p udp --dport domain -j REDIRECT --to-port 53 COMMIT *filter :INPUT DROP :FORWARD DROP :OUTPUT ACCEPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -i lo -j ACCEPT -A OUTPUT -o lo -j ACCEPT -A INPUT -i eth0 -p tcp --dport http -j ACCEPT -A INPUT -i eth0 -p tcp --dport 3127:3128 -j ACCEPT -A INPUT -i eth0 -p udp --dport bootps -j ACCEPT -A INPUT -i eth0 -p udp --dport ntp -j ACCEPT -A INPUT -i eth0 -p udp --dport domain -j ACCEPT -A INPUT -i eth0 -p tcp --dport domain -j ACCEPT COMMIT

I believe that's all, of course this is generally speaking and you'll have to adapt it to fit your needs, but there it is. Just add a dhcp server for convenience and ntp and dns servers so that you don't need to forward those protocols and in order to save more bandwith using the dns server cache and local ntp answers and you're done.

Note that these iptables are dropping all forwarding as in the example the kernel doesn't need to forward anything, squid does it, and for dns and ntp we are using local servers. Of course that if you want to forward some udp traffic, you'll need to add forwarding rules for that.

Transitioning from 0xF6A32A8E to 0xD876D5A3 (moving to stronger cryptography)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1,SHA256 Hi! As Debian is retirying good old 1024 bit keys I'm transitioning to a new 4096 bits key. I still don't have many signatures on my new key, but it is signed with the old one wich to date isn't known to be compromised. My old key was: pub 1024D/F6A32A8E 2000-09-16 Santiago Garcia Mantinan (manty) Primary key fingerprint: 3F0A 12FC 0B55 A917 D791 82D3 72FD C205 F6A3 2A8E My new key is: pub 4096R/D876D5A3 2014-10-06 Santiago Garcia Mantinan (manty) Primary key fingerprint: 06A3 E576 0F61 1B4B B1A9 0E68 B868 8CA3 D876 D5A3 I hope to get this new key on Debian's keyring before the end of the year and hopefully contribute to a stronger keyring. Regards. Santiago García Mantiñán (manty) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlSGNrIACgkQcv3CBfajKo5yRACguwN5wahVJrIli3FGde9KS83W TbYAn2NAf7JZuG9WiiKAX/7DKfc/1J33iQIcBAEBCAAGBQJUhjayAAoJELhojKPY dtWjjE8QAM87bl3z00YgykCGCH7FElqivOesJ3Op16tDU2/o0AP249iA0rRToH9q SR0Ik1oiziWjh7ccUDHmVeIgV2wpso8wcKmJuZbOqQJ4sVvIzRd0IN2G0kBfyFDn +ff/J6aGcDCFLp0nIStEJiycKd4UWcqAoA+RB+wpBwIqH/yXw5l8mEyv7XHwfeHT 3D2r9ocpVdRu9QzElxs8sO7cXOtJ6+wKUurGaDobAIZC/1GIF6UlcAfaV5y0uxYa ZguAye8ff4ggrFxH/dxzTrC/ushIXn7MkjQSIphbkHcpbUwRjXaEBRL49WLnbHpZ lmKfmnUeW8a3zkuTQEfJV3i97k5WHLV/AQpfIbAbTFivXleDBIbf6XIvV3EOVPrr nGN1S625Qj9lrcHpIK1PB8xElqJ31bUss62nlFghaEzEq8eLalJRDMRfOddj/NtN 4Ig7DdJSXUbCjxYVkBPzaRVoTT92sQHJ/c8siH8F5I2+YcwTNTQaOXf3LCfvvKd2 a+x1wc9FlJgi7hrRFiV63MQN68WIDzY2g+irWQSgzLFsA4k4RYGgC+6Ap5R3umln EnBCR6vRmrONS92bb0SuMis1D9WOz62z34OSMNh68Mg1BaWlM2lIczcdTyoIcKLM c0PWnsWTRYjQn+QbAH35YnmB395Z8bNoL2k/XhkuEEVCqXK/hWsm =Tm5k -----END PGP SIGNATURE-----