For the last couple of weeks I have been testing DNS resolvers.
I have been trying to find the best DNS providers for personal use, but mostly for fun and curiosity. Hey, who wouldn't want to know if Quad9 blocks more malicious domains than OpenDNS or whats the provider that has the best porn filter out there.
While doing that, I also setup my own DNS resolvers in the public cloud to test PowerDNS and Bind. All went well, except that they started to be misused and abused as a proxy to DDoS others.
Amplification DDoS attacks
One of the most common DNS DDoS technique is called a DNS amplification attack. Someone, aka bad guy, spoofs the IP address of their victim and issue a DNS query as them. The DNS server responds to the victim IP address, not to the bad guy.
The amplification and danger comes from the fact that the DNS query is small compared to the response. So if the attacker is sending 1Gbps of queries, the victim might be getting 20Gbps of responses. It amplifies the bad guys attack by a big factor. Kinda scary.
To hide their identity, they proxy these attacks through open DNS resolvers, which is what I had for testing. To give an example, this is what I was seeing on my Bind query logs:
03-Jan-2018 20:07:52.493 queries: info: client 191.96.249.xx#34418 (isc.org): query: isc.org IN ANY +E 03-Jan-2018 20:07:52.494 queries: info: client 191.96.249.xx#34418 (isc.org): query: isc.org IN ANY +E 03-Jan-2018 20:07:52.458 queries: info: client 191.96.249.xx#35979 (isc.org): query: isc.org IN ANY +E 03-Jan-2018 20:07:52.792 queries: info: client 191.96.249.xx#35979 (isc.org): query: isc.org IN ANY +E
The bad guy was trying to take down 191.96.249.xx. So he was issuing spoofed queries for "isc.org IN ANY" as 191.96.249.xx. That forced my resolver to send the very large responses that this query generates to the victim IP. Hundreds per second until I shut it down.
Bind, however, has a way to minimize abuse and rate-limit DNS responses per source IP address. After some research, I added the following to my config:
rate-limit { responses-per-second 5; };
That restricted to a maximum of 5 responses per second per source IP Address. That prevents the bad guys from misusing my server for their attacks as 5 requests per second is not enough to take down anyone.
You can read more about DNS amplication attacks here.
Random sub-domains DDoS
However, that was not the end of it. Today, I started seeing a new type of DDoS attack that I still can't find an easy to implement solution on Bind or PowerDNS without parsing logs and banning IP addresses via iptables.
And the victim was Uber.
Someone, likely another bad guy or competitor, started to query hundreds of invalid sub-domains under uber.com from random source IP addresses. That bypasses the rate-limit from my Bind configuration and forces my DNS resolver to contact Uber's DNS servers for each one of those invalid sub domains. This is what it looked like in the logs:
11-Jan-2018 13:01:22.356 queries: info: client 51.15.57.237#37071 (11–149.uber.com): query: 11–149.uber.com IN A11-Jan-2018 13:01:22.357 queries: info: client 51.15.57.237#37071 (15408.uber.com): query: 15408.uber.com IN A11-Jan-2018 13:01:22.357 queries: info: client 51.15.57.237#37071 (146–134.uber.com): query: 146–134.uber.com IN A11-Jan-2018 13:01:22.358 queries: info: client 51.15.57.237#37071 (36850.uber.com): query: 36850.uber.com IN A11-Jan-2018 13:01:22.358 queries: info: client 51.15.57.237#37071 (4399xiaoyouxidaquan.uber.com): query: 4399xiaoyouxidaquan.uber.com IN A
Multiply that by thousands per hour to get an idea. The logs above might be hard to see, but the bad guys were querying random sub domains like those:
12–229.uber.com16419.uber.com246–154.uber.com4399xiaoyouxidayouquan.uber.com49739431z70y5970530741.uber.com5755619.uber.com6831130420.fls.uber.com8ca16.uber.combet165keyiwanma.uber.comaomenguojibqaijiale.uber.combet3651yuleechengkaihuxuzhi.uber.combiopc311.uber.comcdn-m3.uber.comdevpnp.uber.com
Which none of them exist, generating a NXDOMAIN. However, the DNS resolver needs to contact Uber's DNS server for every one of them.
I did a breakdown of the logs and that's how many they were generating in average per hour block:
2018-Jan-11, 1pm: 16 per second2018-Jan-11, 2pm: 56 per second2018-Jan-11, 3pm: stopped2018-Jan-11, 4pm: stopped2018-Jan-11, 5pm: 780 per second2018-Jan-11, 6pm: stopped2018-Jan-11, 7pm: 236 per second2018-Jan-11, 8pm: 412 per second
And that's just from my tiny little resolver in the public cloud. They probably misuse hundreds of open DNS resolvers to generate thousands of fake requests to Uber's network -either to take it down or increase their operation costs.
After that, I restricted my testing resolver to only my IP address to stop the abuse. One thing interesting about this attack is that it has little amplification value. The byte size of 1 query is very similar to the NXDOMAIN response.
However, it can still take them down, while maintaining the anonymity of the attacker — and really put their DNS servers under stress.