It's probably temporary? Move to R53, then figure out what's needed to manage records on two providers (if only for internal processes). Top engineering teams aren't going to knee-jerk this, right? Or did Dyn show some unfixable incompetence?
To be fair Route 53 will split over other providers so you should be good in theory. But yeah if something more specifically targets Route 53 then that could be the same problem.
We use CloudFlare and after this my boss said "set us up with secondary DNS somewhere." Unfortunately, CloudFlare doesn't support being a primary DNS provider with NOTIFY messages. They are designed to handle the DDoS for us by proxying content. It's an interesting problem and I don't know whether to push back to CloudFlare or my boss. Anybody else running secondary DNS after this with CloudFlare?
What you should do depends on your setup and threat model. Do you fear DNS auth going down? Do you think your DNS will be a target? Do you use Cloudflare to hide your HTTP origin IP addresses?
For example, if you fear DNS auth going down, but you must use Cloudflare for HTTPS (say: for caching and SSL certs), then changing DNS off CF makes little sense. You already assume stability by expecting it to work HTTP layer.
If you think you can be a target of DNS attack, I'd say having multiple auth is unlikely to give you more mileage.
If you can afford disabling CF on HTTP layer, exposing your HTTP origin IP and want to have two different DNS auth providers, fine, you can do CNAME. But then you have three vendors to worry about, and problems with each can lead to trouble.
I don't know much about running nameservers but moving to all internally hosted seems like an odd choice to me, can anyone explain whey that's a good move?
With only a modest simplification you can view security as ultimately just being a figure measured in dollars: "it costs an adversary $X to beat these countermeasures." Your goal in securing a system is not to push X to infinity, though that might be a reasonable goal (e.g. if you're a security researcher designing new crypto primitives). Instead your goal in engineering your company's security consists in evaluating the value $V of what you're securing, and then raising X until X > V. There are uncertainties in measuring X and V and in how attackers will view these tradeoffs and so forth, but it's nothing you can't account for by building in an engineering tolerance like X > 2V. The basic story remains.
Spotify simultaneously has large resources and offers a non-essential infrastructure service (music to listen to while you're doing something else). The V gained in DoSing them is very small. They got attacked anyway because they shared infrastructure with other companies, which pools the V together to create something much larger. Some attacker saw a case where V >> X and attacked it to great success until Dyn was able to bring up X again. During the interim, Spotify was down despite having V << X.
In short: Spotify probably can't do DNS better than Dyn, but they can do DNS better than the sort of people who have reason to attack them (presumably trolls, maybe some future hacktivist who doesn't like some business decisions they make, unscrupulous competitors). This attack was a wake-up call for them, "oh, if we're pooling with these other folks then we'll become targets of larger hacktivist attacks and state actors, who are not directly targeting us per se." Those attackers could presumably still take out Spotify's home-rolled DNS, but they have no real motivation to target Spotify in particular any more.
It lower surface attack. With companies like Dyn, they are affected even when someone is targeting other sites, while with internal DNS servers that are only used by themselves they will be down only if someone is attacking them directly.
If someone is targeting them directly it doesn't matter much that DNS is up and running, their site is still down.
"This outage exposed a critical weakness in our DNS hosting configuration. We are taking immediate steps to add additional DNS providers. This should allow us to avoid impact in the future, provided that at least one of our DNS providers is operational."
What happens if you have a DDOS on Route53? I'm sure they can handle the attack, but do you have to pay for the requests? Or are there clauses that they drop the fees if the requests were malicious? If not, the financial risk could easily outweigh the benefits of availability for smaller companies.
I'm in the process of looking for a secondary DNS server for a client but because they rely heavily on geolocation load balancing it's not simple... I wonder if anyone has other recommendation beside UltraDNS for a good slave?
Virtually everyone is behind NAT these days, often multiple layers of NAT. So how does the botnet manage to telnet or ssh into these set top boxes or lightbulbs or whatever? When I want to SSH into my home computer I have to go through elaborate maneuvers to get it to work.
As I understand it UPNP allows a device to negotiate a port map with a NAT box. But the developer of the IoT device would have to specifically want to map port 22 or 23. It's not something that would happen automatically.
So are you saying that these IoT device makers not only hard-coded root usernames and passwords into their devices, but deliberately set up UPnP mappings to those ports?
That looks malicious rather than negligent. Am I missing something?
And so? There is more than one way to spread a botnet.
One particular method is DNS rebinding. They attack your webbrowser, then your browser passes the attack to the device inside the network.
Another means is 'the weakest link'. An insecure and compromised device (even if it just a user account on that device) scans your local network behind the firewall , passes information to a command and control server, which downloads and passes an exploits to the devices it discovers on your network.
So I may be flat out wrong about this, but I believe the clients, once infected phone home, thus (mostly) removing the need for worrying about NAT, as they router will forward the Server's connections to the correct machine once it sees that a prior outgoing connection has been made.
I commented this in another post, but at least for the affected surveillance cameras and recorders, the firmware for these devices is built on Busy Box. The manufacturers either opened up telnet or left telnet open for these firmwares until about 3 years ago when it started getting exploited.
The characteristics for how these surveillance devices were hacked are that the devices are using older firmware with an exploitable telnet feature and that the default credentials are still intact. There are a few vectors (using UPnP, HTTP API, directory traversal) that can be applied to bypass authorization. Throw in a global directory of these devices (shodan.io) and you have the ability to search for these devices with specific firmwares, run every attack vector to compromise these systems, and, once connected, have them do _whatever_you_want_.
> Virtually everyone is behind NAT these days, often multiple layers of NAT. So how does the botnet manage to telnet or ssh into these set top boxes or lightbulbs or whatever?
Infected windows desktop can scan local network and infect IoT devices.
Expect more of this in the near future, single-source infrastructure is becoming a huge liability (not that it wasn't before). I wonder what impact on SLAs it will have when cloud services providers are taken down - will they honor their SLAs or inject DDOS clauses into them to shield themselves. You won't see many standing up to multi-Tbps attacks, at least for the moment.
Doesn't that also provide a huge incentive for cloud offerings to sell DDOS-resistance products and services? Isn't that a huge market already with gigantic margins?
Can any one explain why they keep referring to this as a complex attack? From the article it seems to be a simple volumetric attack. They mention that it uses UDP port and TCP port 53, nothing complex about that...
Am I missing something here. It wasn't an L7 attack (or was it?) Why keep referring to it as complex?
I think the market's expectation is that a DNS provider is prepared for a DDoS of any size, but not necessarily any level of complexity, so that's a lot of incentive to talk up the complexity of the attack.
What's described in this incident report is totally within the capabilities of a single individual with public knowledge, though. If they could have proven otherwise, they probably would have (unless that somehow conflicted with their criminal investigation).
Much of the complexity is likely in building and managing such a large botnet.
There also isn't a lot of details here on the exact nature of the traffic. They say it was hard to distinguish between legitimate traffic and this malicious traffic. So the botnet is at least rotating their requests through lists of customers hosted with them (though that isn't complex, but it is forward thinking. If the botnet was all making non-stop requests for just a few domains, that would be a strong signal to start filtering traffic, first internally, then pushing ISPs to block it upstream).
I wonder what 1.2Tb would do if pointed at a ELB or an AZ in AWS. Someone must have tried at some point but i've never heard of any widespread outages caused by DDOS on AWS. Is it just that bandwidth available to an AWS datacenter is that much bigger than what dyn have?
It's safe to assume that 1.2Tbps isn't a big deal for Google/MSFT/AWS/Cloudflare/Akamai/Yahoo/Verizon/etc.
DNS is normally a low-bandwidth protocol so if you only provide DNS services, needing to purchase 1000x your normal bandwidth to handle these bursts would be miserable. If a DNS provider were also providing video services (Vimeo/Twitch/etc), then a 1.2Tbps increase in traffic could be easily absorbed.
"State actor" today is an abused term, because it helps the victim look less bad, the government will back it up, because it paves way for new regulations and there's no way to prove one way or the other. Especially when it comes to DDoS.
This is insane how big these things are now: "There have been some reports of a magnitude in the 1.2 Tbps range; at this time we are unable to verify that claim."
They mention 100k participating devices/IPs - that would mean an average of 12 Mbps upload. Sounds high, but plausible? [ed: I actually think it's kind of sad that most users aren't on symmetric gigabit links yet... But in this context I guess it's a blessing of sorts... ]
"They mention 100k participating devices/IPs - that would mean an average of 12 Mbps upload"
I doesn't have to be. An attack can be highly asymmetric such amplification/reflection attack. I am not saying this is what happened in the Dyn case but rather addressing your average comment. Just an example having hosts in the botnet send queries to non-Dyn open resolvers on the internet and in those queries they spoof the source IPs of Dyn DNS servers. Now all of those open resolvers on internet start sending return traffic to Dyn. If in those spoof request they request EDNS0 those responses could be up to 4K. Compare that to the size of query packet and you have a lot leverage.
We already know Mirai has been able to reach over 1 Tbps, and we know Mirai was at least one of the cannons hitting Dyn. So 1.2 Tbps is definitely plausible. Mirai has decreased in size to a degree due to more public awareness, but it's still massive.
Readit News
* us-east-1.amazonaws.com: split between internal, UltraDNS, DYN
* spotify.com: all internal nameservers now
* reddit.com: all Route53 now
* github.com: all Route53 now
* netflix.com: all Route53 now
* paypal.com: split between UltraDNS and DYN
No changes made:
* twitter.com: 100% with DYN
At least that's my understanding anyway.
Indeed, if you want the HTTP/HTTPS traffic to go through Cloudflare, the DNS must go through Cloudflare. There are generally two ways to set it up:
a) You move your DNS auth to Cloudflare and allow it to manage it.
b) You keep managing your domain yourself, and CNAME to Cloudflare. See: https://support.cloudflare.com/hc/en-us/articles/200168706-H...
What you should do depends on your setup and threat model. Do you fear DNS auth going down? Do you think your DNS will be a target? Do you use Cloudflare to hide your HTTP origin IP addresses?
For example, if you fear DNS auth going down, but you must use Cloudflare for HTTPS (say: for caching and SSL certs), then changing DNS off CF makes little sense. You already assume stability by expecting it to work HTTP layer.
If you think you can be a target of DNS attack, I'd say having multiple auth is unlikely to give you more mileage.
If you can afford disabling CF on HTTP layer, exposing your HTTP origin IP and want to have two different DNS auth providers, fine, you can do CNAME. But then you have three vendors to worry about, and problems with each can lead to trouble.
I don't know much about running nameservers but moving to all internally hosted seems like an odd choice to me, can anyone explain whey that's a good move?
Spotify simultaneously has large resources and offers a non-essential infrastructure service (music to listen to while you're doing something else). The V gained in DoSing them is very small. They got attacked anyway because they shared infrastructure with other companies, which pools the V together to create something much larger. Some attacker saw a case where V >> X and attacked it to great success until Dyn was able to bring up X again. During the interim, Spotify was down despite having V << X.
In short: Spotify probably can't do DNS better than Dyn, but they can do DNS better than the sort of people who have reason to attack them (presumably trolls, maybe some future hacktivist who doesn't like some business decisions they make, unscrupulous competitors). This attack was a wake-up call for them, "oh, if we're pooling with these other folks then we'll become targets of larger hacktivist attacks and state actors, who are not directly targeting us per se." Those attackers could presumably still take out Spotify's home-rolled DNS, but they have no real motivation to target Spotify in particular any more.
If someone is targeting them directly it doesn't matter much that DNS is up and running, their site is still down.
The question you should ask is why did these companies used an external DNS in the first place?
(And I know it's not that simple, but that's probably the basic reasoning behind it.)
On previous 3 Saturdays they lost between 40 and 60 domains.
https://status.heroku.com/incidents/965
"This outage exposed a critical weakness in our DNS hosting configuration. We are taking immediate steps to add additional DNS providers. This should allow us to avoid impact in the future, provided that at least one of our DNS providers is operational."
Shame on reddit, github, and netflix for learning literally fuck-all from this.
Virtually everyone is behind NAT these days, often multiple layers of NAT. So how does the botnet manage to telnet or ssh into these set top boxes or lightbulbs or whatever? When I want to SSH into my home computer I have to go through elaborate maneuvers to get it to work.
So are you saying that these IoT device makers not only hard-coded root usernames and passwords into their devices, but deliberately set up UPnP mappings to those ports?
That looks malicious rather than negligent. Am I missing something?
(Well, not surprised as much as disappointed.)
One particular method is DNS rebinding. They attack your webbrowser, then your browser passes the attack to the device inside the network.
Another means is 'the weakest link'. An insecure and compromised device (even if it just a user account on that device) scans your local network behind the firewall , passes information to a command and control server, which downloads and passes an exploits to the devices it discovers on your network.
The characteristics for how these surveillance devices were hacked are that the devices are using older firmware with an exploitable telnet feature and that the default credentials are still intact. There are a few vectors (using UPnP, HTTP API, directory traversal) that can be applied to bypass authorization. Throw in a global directory of these devices (shodan.io) and you have the ability to search for these devices with specific firmwares, run every attack vector to compromise these systems, and, once connected, have them do _whatever_you_want_.
And that's only on the default port.
Deleted Comment
Infected windows desktop can scan local network and infect IoT devices.
Expect more of this in the near future, single-source infrastructure is becoming a huge liability (not that it wasn't before). I wonder what impact on SLAs it will have when cloud services providers are taken down - will they honor their SLAs or inject DDOS clauses into them to shield themselves. You won't see many standing up to multi-Tbps attacks, at least for the moment.
Am I missing something here. It wasn't an L7 attack (or was it?) Why keep referring to it as complex?
What's described in this incident report is totally within the capabilities of a single individual with public knowledge, though. If they could have proven otherwise, they probably would have (unless that somehow conflicted with their criminal investigation).
There also isn't a lot of details here on the exact nature of the traffic. They say it was hard to distinguish between legitimate traffic and this malicious traffic. So the botnet is at least rotating their requests through lists of customers hosted with them (though that isn't complex, but it is forward thinking. If the botnet was all making non-stop requests for just a few domains, that would be a strong signal to start filtering traffic, first internally, then pushing ISPs to block it upstream).
1. Device backdoor open Port 23 (telnet), used to take over loT devices.
2. The loT devices attacked through Port 53 (DNS).
DNS is normally a low-bandwidth protocol so if you only provide DNS services, needing to purchase 1000x your normal bandwidth to handle these bursts would be miserable. If a DNS provider were also providing video services (Vimeo/Twitch/etc), then a 1.2Tbps increase in traffic could be easily absorbed.
[p.s.] The question is precisely this: is it possible that a DDoS attack on DNS can be used to affect/mask a MITM attack.
I doesn't have to be. An attack can be highly asymmetric such amplification/reflection attack. I am not saying this is what happened in the Dyn case but rather addressing your average comment. Just an example having hosts in the botnet send queries to non-Dyn open resolvers on the internet and in those queries they spoof the source IPs of Dyn DNS servers. Now all of those open resolvers on internet start sending return traffic to Dyn. If in those spoof request they request EDNS0 those responses could be up to 4K. Compare that to the size of query packet and you have a lot leverage.