[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 01 Jan 2012 20:54:49 -0700
From: Kurt Seifried <kseifrie@...hat.com>
To: oss-security@...ts.openwall.com
CC: Solar Designer <solar@...nwall.com>
Subject: Re: speaking of DoS, openssh and dropbear (CVE-2006-1206)
OpenSSH and Dropbear
Attack of the zombie clients
So an old problem has reared its ugly head. Many services, especially those that require authentication will hold a new connection open for some time in order to give the client time to authenticate. If a user is expected to type their password in then the amount of time the connections must be held open for needs to be on the order of tens of seconds if not a minute or more (especially if you use two factor authentication or other things that require the user to take some action). This is especially true for those that support cryptographic connections such as SSH and anything SSL or TLS enabled incur expensive computation, especially on the server.
By establishing a large number of connections and holding them open (this typically requires only a few packets or less per second from the attackers system) an attacker can hold a connection open indefinitely or until a timeout condition is reached.
We have seen this attack before: SYN flooding, Slowloris, airline seat reservations, email spam, password brute forcing attacks, etc.
There are several general classes of mitigation and response to these attacks (please note that I present several which won't apply to OpenSSH, but I feel we should think about them as this class of problem affects services other than OpenSSH):
1) Early drops
By dropping a connection early the server can reduce workload and make resources available for legitimate clients. The downside of this is that services that require some form of interactive authentication (e.g. SSH with a password) rely on a human to type the password in which typically takes a few seconds. In this case early drops are more likely to affect legitimate clients rather than attackers.
2) Random drops
Another strategy is to randomly drop connections when load gets high. The theory is that legitimate connections will continue to try and connect, by dropping connections and reducing load there is a better chance that legitimate clients will get through. Unfortunately with a typical DoS or DDoS style attack to attackers connections will vastly outnumber legitimate connection attempts.
3) Progressive timeouts/lockouts
To prevent password guessing attacks at the console some Linux systems implement a progressive timeout, if you get the password wrong you have to wait a second, if you get it wrong again you have to wait 2 seconds, then 4 seconds, etc. This reduces the number of attempts an attacker can make. In terms of OpenSSH we could for example implement a lockout, if you connect but fail to login you have to wait 10 seconds, the second time you need to wait 20 seconds, etc. There are a large number of downsides to this however: you need to store a database of IP's or networks to lockout and for how long, you may have multi user systems connecting to yours which means one user making a typo (or maliciously fail to login, thus locing everyone else out).
4) Reputation based systems
Reputation based systems are another strategy to deal with attackers. Email uses this more than any other protocol, largely due to the semi-centralized nature of email (you typically send email to a local server that then sends it to the remote end's server as opposed to initiating a direct connection from your workstation to the remote server). Databases like ORBS and so on attempt to maintain a comprehensive list of known "bad" systems and networks that you probably don't want to accept email from. The problem with these (especially for things like SSH and web clients) is two fold: you end up playing whack-a-mole with the bad guys and you can't simply block connections from residential ISPs for example since there is often a legitimate need for administrators/users/etc to login from home.
However approaching reputation based systems from a white listing view may work, for example if an IP address successfully connects and authenticates that IP can be stored, and in future it can be given preference over unknown IP addresses. This could be extended, for example by doing a WHOIS look-up on the IP and white listing the entire network that IP is from (thus for clients with dynamic IP they aren't constantly having to get back on the white list).
5) port knocking
Another possible approach is to add a transparent layer of authentication/firewalling in front of the server. Port knocking simply put makes the client send a packet to either a specific port # (e.g. port 47362) before they can access port 22, or they must send a packet with specific contents (e.g. a password/shared secret). One thought for longer term research here is to combine port knocking with the TCP-IP handshake, send back a nonce and the client must hash the nonce+shared secret and reply with it in order to gain access to the desired port. This of course would require a retrofit of existing TCP-IP stacks/applications, you'd have to store a list of shared secrets/passwords and the hosts/networks associated with them and so on, but is included because I think it's interesting =).
6) hash cash and work functions
The idea here is simple, make the attack computationally expensive and the attacker will not be able to cause as much trouble. For large scale or sustained attacks simply increase the work function (like bcrypt, you can effectively select how much work it is to decrypt the password), this could also be done in the fly, as an increasingly number of connections occur increase the work function.
7) Grey listing
This is more applicable to email but can work with other protocols potentially. Basically you tell clients that their connection cannot be serviced and that they should come back later and try again. Legitimate clients will come back and attempt to connect again, most illegitimate clients (at least in the spam world) will simply move on and try other hosts. This type of solution will probably not work well with interactive protocols such as SSH or for targeted attacks.
8) Syn cookies
While specific to TCP-IP some protocols can lend themselves to similar approaches. The problem with SYN flooding attacks was that the attacker simply forged a lot of SYN packets, creating thousands of partially opened connections on the victims system. SYN cookies exploited the fact that TCP-IP sequence numbers obeyed certain rules, by specially crafting the sequence ID the server could then drop the locally held connection, if the client replied legitimately then the server could reconstruct the information needed for the connection based on the information in the packet sent by the client.
9) VPN connections
Connections to sensitive services can be proxied via VPNs, but in this case, where we are worried about DoS/DDoS attacks that only moves the problem elsewhere (to the VPN server, most of which can be attacked in a similar manner). I include this for completeness however.
10) make the client prove they are legit
This approach is taken widely on the web, CAPTCHAs, JavaScript, Flash and so on can all be used to weed out bots and automated clients, giving preferential treatment to legitimate web clients (well, at least those not running NoScript/etc.). Unfortunately with OpenSSH this solution isn't overly applicable.
-------------
But what if the attacker has a 10,000 node bot net?
I think at this point the only realistic solution is some sort of hash cash with a heavy workload and a reputation based / white listing system, so once you successfully authenticate from a given IP or network that IP or network gets preference for future connections. Port knocking could also potentially work in this situation. This of course fails if the attacker has the same IP as you (e.g. a NAT'ed wireless network) or is on the same network as you are.
-------------
The first three solutions (early drops, random drops and progressive timeouts) can be implemented in OpenSSH server side without requiring any changes to the protocol or clients.
Reputation based systems can be implemented within OpenSSH (but I wouldn't, because it is a bad bad idea) or at the OS level in firewall code, reputation based systems would require no changes to OpenSSH clients or the protocol.
Port knocking can implemented at the OS or firewall level, but requires client software and user training/etc. Widespread deployment isn't going to happen anytime soon (although I'd be curious to see how adoption would be if we shipped something to support it by default).
The rest of the solutions do not lend themselves to this problem or would require significant changes to the OpenSSH protocol/client/server which is a bad bad idea.
Anything we do to address this issue should be extremely simple and conservative, the OpenSSH server and client are very stable and robust pieces of code, any modifications to them make me nervous.
I suspect the simplest and more effective solution might be some form of progressive timeout for IP's that fail to authenticate (drop the connection entry silently and ignore them in favor of real clients).
Long term I'd like to see more work on hash cash type solutions, being able to arbitrarily set or have a reactive system that requires increased work on the client end to prove they are a legitimate client would help with this whole DoS/DDoS class of problem to some degree.
--
-- Kurt Seifried / Red Hat Security Response Team
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
