DDoS: WTF

Maybe some online stuff you're using has been down, or performing poorly, and you're hearing it is, or might be, DDoS.

What does that mean, and why is it hard for them to solve this problem?

DoS stands for Denial of Service - straightforward, an attacker is trying to deny (prevent) someone the ability to use a service. Why? We'll get to this at the end of the article when we talk about how to make it stop - it'll make sense then.

(Where did the other D go? Hang on, it's coming.)

While DoS can be pretty complex, we can simplify it into two basic categories - causing crashes, and resource exhaustion.

To cause a crash, you send the target something that it can't handle (like a malformed HTTP request) gracefully, and it crashes or hangs. These used to be common, but AFAIK, they are not anymore - most Internet-facing software has got better at handling bad input without crashing. So we won't focus on these.

Resource exhaustion takes many forms. You can try to fill up your target's bandwidth. You can try to cause them consume all their CPU, RAM, or disk. You can even try to get them to run out of the less obvious "hidden" resources like OS file descriptors, database handles, socket buffers, that kind of stuff. The basic game is to get your target to use up something that they need to serve actual requests.

If the attacks are coming from a small number of sources, this is usually fairly easy for a service to get under control (though harder for, say, some random person whose home network is being targeted.) You identify the source(s) via IP address and block them via a firewall, via special rules in your web application, whatever. So long as they are not filling up the network bandwidth that people use to get to you, this is pretty effective.

(Side note: There's a variant here that's sometimes called a "self-DoS", and it's not that uncommon. Maybe you make a mistake in your application, and certain things consume a lot more server resources than you expected. It's also not that unusual to have systems that are, internally, very aggressive about retrying when failures occur, so when one does, you end up hammering your own infrastructure with retries.)

This is where the other D, Distributed, comes in: if you are getting hit by many different sources, defense gets a lot harder! You can no longer easily identify and block attackers by IP address, and you end up in a never-ending game of trying to distinguish which traffic comes from real users, which you want to serve, and attackers, who you want to reject. It's overwhelming.

DDoS usually comes from botnets: networks of computers that have been compromised, or have unwittingly opted in to participating (see: Samsung Smart TVs) - there's a whole interesting article there, but I'm going to have to skip it for now.

So, what can you do about a DDoS? You have basically four choices:

Increase capacity

Option 1 is to increase your capacity so that, despite the attacker using up a bunch of your resources, you still have enough to serve actual users. Whether this is feasible depends a lot on how your service is designed and deployed. If you're running on your own servers in your own datacenter or point of presence (POP), this can mean buying and deploying actual equipment, so we're not talking "quick" here. If you're in a cloud, you might be able to quickly pay to increase your bandwidth, bring up more VMs or containers, whatever's going to get you what you need. But, your application needs to be already designed to scale in this way.

But let's be clear, this option really sucks. Getting more resources costs money. Something that was "just" a technical attack has now become a financial attack. If the attacker's goal is to knock something offline permanently, making their services unnecessarily expensive to run can drain companies' bank accounts, or make it just not worth doing for individual or small groups. In fact, this is very much a known subgenre of cyberattack: you go after a target who you know is going to automatically scale up, run up their AWS bill, and bankrupt them.

Decrease your resource needs

Another option is to try to make the attack less painful by making each request consume fewer resources. There is a very wide variety of techniques here, from optimizing your code to temporarily turning off features, and more. This is very specific to the individual service, and options are not quick (optimizing code) and/or need preparation ahead of time (having simple knobs to turn off features). I'm not going to talk about it a lot, but I wanted to include it for completeness.

Shed load

This is the main strategy that services use: try to reduce resource usage by serving users, but not serving attackers.

One way is through what I mentioned above, trying to figure out which IP addresses are real users and which are attackers. This is hard, imperfect, and, as mentioned before, something that DDoS aims to make impossible.

An option that you've definitely seen is CAPTCHAs or proof-of-work systems like Anubis. The goal here is to serve up a page that's quick or cheap for you, but uses resources (compute resources in the case of Anubis, or human eyeballs and reasoning - or relatively expensive AI - in the case of CAPTCHA) for the client. There's a lot of complexity here, and I can't go into all of it (in part because I don't know it all!), but the basic idea is to flip the asymmetry: normally, the attacker is sending you a request that's cheap for them to send, expensive for you to serve. By making everyone visiting the site do some work, you hope to make things expensive, resource-wise, for the attacker, while keeping the work bearable for real users. But, people find this annoying, and there are accessibility issues if you are not careful (and even if you are), and attackers with a lot of resources might be able to deal with it anyway. This method also doesn't work well or at all for APIs and other machine-to-machine communication: the whole point in this case is that there is not a human sitting there to solve your CAPTCHA.

Another is by using authentication: if real users are able to log in, you can, temporarily or permanently, depending on the service, reject any requests that don't come with a valid login cookie or token. This is nice, because it draws a clean line between "people authorized to use this service" and those who aren't. But it has drawbacks. Many services want to be able to serve users who are not logged in. Real users need to be able to log in in the first place, and maybe the DDoS is preventing them from doing so. And even just properly checking authentication tokens involves cryptography and/or database lookups, so you're still having to consume significant resources before you can decide who to let in.

The last one that I'm going to talk about in this section is to outsource the problem: pay someone else to deal with it. The idea here is that you put some other service in between you and your users, and it's that service's job to (a) serve your static content and/or (b) figure out who the attackers and real users are, and only send traffic from real users to you. There are lots of offerings in this category, but, in general, (a) is called a CDN (Content Distribution Network) and (b) is what Cloudflare (and others) are famous for. To focus specifically on Cloudflare: their job is to have tons and tons of bandwidth, be really good at dropping traffic, and be experts at identifying botnets. If they're doing their job well, they are serving up your static assets from their own CDN, and for stuff that's dynamic, only forwarding traffic to your service if they're pretty sure it's not an attacker.

You can see, I'm sure, why this has become an attractive option. By some estimates, Cloudflare sits in front of something like a quarter of websites these days. But it also has downsides, of course. This makes Cloudflare itself a potential point of failure. To do their job, they need (well, it's complicated, but for simplicity I'm going with "need") to be able to snoop on your users' traffic, and impersonate your website - after all, the whole point is that they pretend to be your website or service! They're a business, and while they do offer a free tier, if you're a significantly large site, it costs money. My understanding is that it's not as good for APIs, because its most powerful detection and mitigation mechanisms work best with humans, and streaming stuff through a third-party proxy can be iffy.

Get them to go away

The real end goal is to get the attackers to go away, of course: by mitigating the attack so that it's ineffective, by outlasting them, or ... through other means. But what it takes to make them go away depends on why they're doing it.

So why are they doing it?

So, getting a DDoSer to go away ends up as a game of money and patience: some will go away on their own because they made their point, got bored, moved on to another target, etc. Others will persist so long as they are being effective: as long as they're causing problems for someone they're mad at, they'll keep going; making the attack ineffective may cause them to give up, though if they are trying to waste your money, they might continue anyway. Possibly, they'll know you're spending time and money to keep them at bay, and will keep it up anyway until you get exhausted and pay them.

(I should add "hacking back" and "law enforcement" as ways to get them to go away for completeness, but the former involves skills most don't have, and the latter takes a very long time, if it's effective at all.)

When your favorite site is down due to DDoS, spare a thought for the administrators who have to understand and mitigate it; maybe they'll write up an incident report afterwards to tell you what happened - many, but not all, do. Most of these mitigations are fairly effective even if attackers know about them, but not all, and they might prefer not to take the chance. And if they paid an attacker to go away? Well, they don't want others to know they can be extorted, so they'll probably never say.