Is it confirmed yet that so-called IoT devices were the bots?
Bruce was on point if so, arguing a couple weeks ago that accountability needs to happen on the manufacturers:
"What was new about the Krebs attack was both the massive scale and the particular devices the attackers recruited. Instead of using traditional computers for their botnet, they used CCTV cameras, digital video recorders, home routers, and other embedded computers attached to the Internet as part of the Internet of Things.
Much has been written about how the IoT is wildly insecure. In fact, the software used to attack Krebs was simple and amateurish. What this attack demonstrates is that the economics of the IoT mean that it will remain insecure unless government steps in to fix the problem. This is a market failure that can't get fixed on its own.
I feel like I hadn't thought of this as a market failure until reading your post calling it that. You're absolutely right about it. That's exactly what it is and the need for government involvement is quite obvious now. Suppliers are going to need to be held liable for the negative externalities their product offerings create, otherwise we're stuck at an equilibrium point where this situation does not improve.
If ISPs were treated like a utility and charged per bit, customers would have an incentive to ensure that their devices weren't dumping traffic onto the internet. It's rare that you can see a dashboard showing your usage, even rarer to see a dashboard showing your usage, broken down by device.
You can call that "getting the government involved" but it's allowing suing for damages due to negligence, which is a fairly basic form of involvement, the sort of thing at the base of the market to begin with. That is to say, it's a bit strange to call this a market failure, because the market will (imo) take care of it once you can assign liability.
Nope. Many of these compromised routers and webcams are not based on U.S. soil, so they're outside of U.S. jurisdiction. But even if some enterprising lawyer could attach a legal claim to them, most of these guys are tiny, and while you could easily sue some individual companies out of existence, it would not have much impact on the broader problem.
That's probably too distributed a set. You'd have to hit the device manufacturers (say, ARM or Intel), or vendors (Amazon). Hold them liable for problems.
Hit the distribution channel and I suspect you'll see a rapid increase in accountability and security measures.
Hopefully it's not related threats about hacking during the election.
Remember that recently Biden openly threatened cyber attack on Russia if they make any attempt to tamper with the election. Which is completely unprecedented, as is the notion that DOD is openly saying Russia was behind DNC and other attacks.
Also what amazed me is that he would casually threaten to strike Russia. It seems that no one considers these attacks as an act of war. But that's what they are.
Thank you, missed this piece but it was interesting.
I disagree with him on the point of "Who would do that?" He might be right about state level actors, but I think he discounts the motivations of crazy/disillusioned people, bored and curious people, and especially teenagers.
When I was a teenager, the Internet wasn't a thing yet, but we sure dreamed of all kinds of crazy schemes for taking out the phone company, power, anything really. We talked about anarchy and many "taboo" topics I can't mention here. The thing is we were good kids at heart and we had the discretion and morals not to act on those things. All of this happened in a time where our instant communication was the phone or meeting up in person. Today, it is infinitely easier to seek out like-minded people and to replace those who drop out. The ability to seek out confirmation and push is easier than ever as well.
Unfortunately, there are plenty of people that don't have that. Just because someone is a misguided teenager or crazy person does not mean they do not have intelligence, organization, and skills. Many of us certainly did our share of things and had the power, but I wonder what might have happened if we didn't stop ourselves in some cases. While perhaps the organization and probing nature likely hints at something else, it's really not that unusual for people to just mess around. Some people as they say also just want to watch the world burn. A couple of rough years in my teens, I certainly felt that way at times. I did plenty of things I'm not proud of, many people just have no shame and will take it that much further.
In the end I probably agree in terms of who is most likely, but I am kind of surprised that there were not more possibilities mentioned. Even 20 years ago, attacking Internet infrastructure seemed an obvious thing to do to us and we used to love talking about fun ways to ruin things over a burger at lunch. I mean is it really that hard to fathom people would think about attacking targets other than some organization, government, or other kind of company's servers?
Schneier's post is hardly prophetic. The idea that "china is attacking the internet" is so well ingrained, that this 2-year-old fake security attack map has "china mode", to make most of the attacks seem to come from China (part of the mockery of such maps): https://github.com/hrbrmstr/pewpew
> "But technology providers in the United States could suffer blowback. As Dyn fell under recurring attacks on Friday, Mr. York, the chief strategist, said such assaults were the reason so many companies are pushing at least parts of their infrastructure to cloud computing networks, to decentralize their systems and make them harder to attack."
Pushing your infrastructure to cloud computing is not decentralization - it's centralization, and we're all doing it. Imagine if an attack like this was against AWS... we'd all be screwed.
Interestingly, in some ways this is a big selling point of AWS/Azure/Goog. The absolute scale they can handle is way up there.
The downside of course, is that whilst their infrastructure can likely handle it, handling the bill associated with 'just scale up your service' could be worse than the attack itself.
AWS has considerable defenses against DDOS attacks of all types - here's the video from Reinvent 2015 which introduces many of Amazon's defenses as well as best practices - https://www.youtube.com/watch?v=Ys0gG1koqJA
Interestingly, the presenter notes that Amazon had seen a drop in DNS as an attack vector in 2015. I asked the presenter (Product Manager) why they hadn't productized the DDoS attack dashboard so you could be aware if you were being attacked (and it was being absorbed by AWS) and his response was that there was insufficient demand at that point to justify the developer staffing. He gave me his card and asked to request the feature so he could us it to make the case internally.
Does anyone here have stories of being successfully DDoS'd on AWS (other than by their own staff :) ?
If Azure and Google would like to gain a competitive advantage over AWS, then I would suggest this: Build out a suite of tools for fighting DDOS. Enable private consultants and companies to provide this as a service. Do this in such a way, that cloud customers save money and have to worry about less. Hell, let companies jump in structured as insurance companies! Also bring in cooperation with law enforcement and use data gathering to catch and prosecute DDOS-ers.
Yup until this morning, AWS was using Dyn as the sole provider of nameservers for the us-east-1 zone. So this attack did have a pretty substantial impact on some AWS services until they updated us-east-1 to use the more diverse set of nameservers their other datacenters use.
so it will be eventually Cloud VS DDoS eh, both can scale indefinitely so the limit is money, which makes the DDoS guys wins, they practically stole CPU/RAM/NET where cloud providers need to buy hardware as usual
Unless we can somehow secure every net-connected devices, ha (I don't know whether to cry or laugh right now)
You're correct, it's centralisation, at least for the whole community.
It decentralises that one company's DNS -- instead of having one or two DNS servers, perhaps at two sites, they now have 20, at 20 sites. If someone wants to target them, they're probably better protected.
But it's the same 20 servers as a million other companies, so the chance of those servers being a target is much greater.
The cloud can be more decentralized but it more expensive, Done properly having redundancy across multiple clouds aws, rackspace, google, azure, in geographically different areas with different internet service providers it can be done in a very distributed decentralized fashion, just no one actaully does that. Instead they throw everything on one provider and pray its is backed up and secured by that cloud provider better than the IT guy down the hall they just laid off.
This is one of the many reasons AWS and cloud computing in general are way overrated.
I know of a company that pays an AWS bill sufficient to buy the equivalent of their pre-cloud datacenter's hardware every 1.5 months. The extra staff required to perform hardware maintenance would also cost about 2 months' worth of AWS each year (that means they're paying ~3x more than they would with hardware). Yet they moved to the cloud because it's the hip thing to do.
Cloud has upsides and things that are useful, especially for smaller proprietors who can take advantage of cheap droplets from DigitalOcean et al, but for grown-up companies, moving off your hardware shouldn't be automatic.
I think in some cases it might simply be the means to dump 1) people/groups that just don't have a large scale mindset and 2) bypass business processes that are absolutely not designed for large scale systems.
In that scenario you have a bunch of entrenched groups fighting about capex, capacity planning and budget all to get barely enough hardware to account for what you're doing in the next 3-12 months. Instead of taking a step back and creating a long term simple process for regular growth and replacement they get caught in the weeds because they have very old school mindsets.
Then you have your old school finance groups who are using terrifyingly delicate and complex interconnected spreadsheets to manage hardware expenditures and depreciation while maintaining old school draconian policies concerning CapEx budgets but allowing you to basically go nuts with OpEx.
You could try to change the culture in these entrenched groups who will view your attempts to make things better as political moves against them or you could just say "we're moving everything into the cloud" and make a complete end run around all of the people and baggage. The former is probably the "right" thing to do but the latter is going to let you focus on your product letting you get you back to being competitive.
There's also the difference between cabled systems, in which multiple elements can independently support load, and chained systems, in which any given link can fail.
The BBC was affected by the Dyn outage not because they themselves relied on Dyn, but because components of their site did.
I fully agree with you about the paradox of how, in the intent to de-centralize we centralize into cloud VPSes and managed services.
The real reason for the move is that same showtune that we keep hearing in our heads and wish we could tune it out: it's cheaper to move from physical infrastructure to the cloud. It's cheaper to skimp on security by not updating IoT devices. It's cheaper to skimp on security because features need to come first. It's cheaper to outsource operational management to parties with less expertise in places that pay less. To spend less time securing infrastructure perimeters because it costs money.
We feel almost as if we feel comfort hiding behind heavyweights like Google and Amazon will protect us from the bad elements of the world, where we hear about major breaches every few weeks (eg., Yahoo being the most recent). Will this strategy pan out long-term?
With this DDOS, articles about machine learning picking up better password-cracking/guessing algorithms by having previously analyzed large volumes of passwords, major breaches in the financial world, talk of state-sponsored attacks (a la DNC emails) it certainly FEELS like the Internet has gotten a little bit more wild.
AWS was hit today, we saw a spike failures. Got hold of one of AWS guys and they basically noticed that the issue they saw in the US earlier in the day happened again in EU west. Funnily enough they probably could have avoided it if they'd deployed their mitigation to the other zones.
I'm pretty sure DDos against http resources have become quite hard to pull of, which is why there was string of attempts to blackmail smaller email provider but nothing like it happens to similar startups relying on the web. Even the Linode attacks are only possible because they're highly target at a few critical systems there.
We seem to be needing more concerted action on what is a consumer minimum standard for an internet connected device.
Consumer devices have to be more secure because if the low user skill level - and interest.
I am always reluctant to say "there should be a law against it" but frankly if we cannot mandate minimum standards of uogradbility and security for devices we will just keep handing over our devices to the first person to scan them.
Or you need to make it easier for the 'black hole' solution to be pushed further and further back to the sources of the bad traffic.
A remote site shouldn't be able to get you banned from the Internet (by it's self); but it MUST be able to say, "This host is being abusive, restrain them from sending me data". ISPs SHOULD use that information to evaluate if a host from their network might be compromised or otherwise a negative player. ISPs SHOULD also take steps to inform, and link to educational resources, customers which are being bad citizens of the Internet. ISPs SHOULD also be financially motivated (punishments to them) for allowing too many uncivil customers online; this might take the form of instead banning that ISP from the Internet as a whole.
So, as your ISP, I'm going to be held responsible for the actions of you, my customer/user?
Okay, if I'm going to be liable, financially or otherwise, well, then we're gonna have to make some changes around here.
First off, I'm going to have to heavily filter and restrict what traffic you can send out to the Internet. What isn't filtered or restricted is going to have to be inspected, logged, and retained for a period of time.
Next, because I can't be certain that you're RFC3514 compliant and that at least some of the bits you're sending aren't malicious, I'm going to have to prevent you from sending out any encrypted traffic. Instead of allowing you to use any DNS servers you want, you're going to have to use mine (DNS is heavily abused for DDoS attacks). Outgoing e-mail will be automatically redirected to my internal smart host (STARTTLS will be blocked, by the way) and I'm gonna have to log, read, and retain it all. HTTP traffic will be transparently proxied and all requests and responses will be logged and retained.
That's just the beginning. Are you sure this is what you prefer as your "solution"?
As a network operator, I believe that your ISP should be nothing more than a dumb pipe and allow the bits that you send to pass through freely. As an ISP customer, that's how I want my ISP to act. (If something gets reported or I "notice" you for some reason then, sure, I'll look into it. Otherwise, I try to fuck with my customer's traffic as little as possible.)
I'll agree that there is certainly a problem, but it is not because of ISPs.
> this might take the form of instead banning that ISP from the Internet as a whole.
I agree with some of your points, but fracturing the internet is not a viable option. It might make sense if it were a healthy, competitive market instead of the near monopolies that exist today. Imagine banning Comcast, or AT&T.
It's controversial, but I kind of agree. You need FCC approval to broadcast a radio signal due to the risk of interfering with other traffic, and you should have FCC approval that your IOT device meets minimum security standards before being sold.
Why rely on end devices? The infrastructure itself should be designed so that it cannot be broken that easily. Maybe we should return to metered connections, maybe we should implement a protocol to control routing.
The Internet has grown without proper planning using a lot of "quick and dirty" hacks (for example NATs, peering agreements) and today we just see the result. It reminds me of poorly designed email protocols that resulted in spam being the biggest part of email traffic.
I'd say [i]the Internet has grown using a lot of "quick and dirty" hacks [/i]
If internet should wait until all use cases were created, it wouldn't exist. It's power was exactly that people could think on how to create things on top of was available. Many RFCs came afterwards.
The standards don't need to be raised much. Banning the sale of internet-connected devices with non-random default passwords doesn't seem too intrusive for the benefits it will bring.
It's fashionable to blame Russia these days, but what country manufactures the most IoT devices, and has the type of government that could mandate backdoor access?
I think what the OP is implying is that these static admin passwords were put as a deniable backdoor. If it was a Chinese gov scheme it is quite clever as a real backdoor would have been obvious, while this just looks like total incompetence.
> It is too early to determine who was behind Friday’s attacks, but it is this type of DDoS attack that has election officials concerned. They are worried that an attack could keep citizens from submitting votes.
> Thirty-one states and the District of Columbia allow internet voting for overseas military and civilians. Alaska allows any Alaskan citizens to do so.
I had no idea any states allowed voting online. I wonder if the general population will ever get access to that.
If they're absent T ballots, they're not counted until several weeks later (unless the total amount of absent T ballots is larger than the margin between any candidate to ballot measure).
This seems so out of the blue, the last attack was targeting krebs for exposing extortionists. Who is being attacked this time and why?
There is a lot of talk of iot botnets but little to no evidence. This seems too vague and up in the air.
If all it takes is script kiddies and random extortionists to generate such large 1 Tbps scale attacks then we appear to be reliant on an unbelievably fragile base.
There is a growing realization of the need for more decentralization of services but these kind of attacks is going to drive more centralization if only Google scale companies can manage to stay up. I think this is drop everything and fix time for the IT profession.
If their claim is true, does anyone think, it will turn many sympathizers against them? I don't think attacking normal bushiness is a good thing to do.
Readit News
Bruce was on point if so, arguing a couple weeks ago that accountability needs to happen on the manufacturers:
"What was new about the Krebs attack was both the massive scale and the particular devices the attackers recruited. Instead of using traditional computers for their botnet, they used CCTV cameras, digital video recorders, home routers, and other embedded computers attached to the Internet as part of the Internet of Things.
Much has been written about how the IoT is wildly insecure. In fact, the software used to attack Krebs was simple and amateurish. What this attack demonstrates is that the economics of the IoT mean that it will remain insecure unless government steps in to fix the problem. This is a market failure that can't get fixed on its own.
"
https://www.schneier.com/blog/archives/2016/10/security_econ... ("Security Economics of the Internet of Things")
Hit the distribution channel and I suspect you'll see a rapid increase in accountability and security measures.
Remember that recently Biden openly threatened cyber attack on Russia if they make any attempt to tamper with the election. Which is completely unprecedented, as is the notion that DOD is openly saying Russia was behind DNC and other attacks.
[1]: https://news.ycombinator.com/item?id=12764898
I disagree with him on the point of "Who would do that?" He might be right about state level actors, but I think he discounts the motivations of crazy/disillusioned people, bored and curious people, and especially teenagers.
When I was a teenager, the Internet wasn't a thing yet, but we sure dreamed of all kinds of crazy schemes for taking out the phone company, power, anything really. We talked about anarchy and many "taboo" topics I can't mention here. The thing is we were good kids at heart and we had the discretion and morals not to act on those things. All of this happened in a time where our instant communication was the phone or meeting up in person. Today, it is infinitely easier to seek out like-minded people and to replace those who drop out. The ability to seek out confirmation and push is easier than ever as well.
Unfortunately, there are plenty of people that don't have that. Just because someone is a misguided teenager or crazy person does not mean they do not have intelligence, organization, and skills. Many of us certainly did our share of things and had the power, but I wonder what might have happened if we didn't stop ourselves in some cases. While perhaps the organization and probing nature likely hints at something else, it's really not that unusual for people to just mess around. Some people as they say also just want to watch the world burn. A couple of rough years in my teens, I certainly felt that way at times. I did plenty of things I'm not proud of, many people just have no shame and will take it that much further.
In the end I probably agree in terms of who is most likely, but I am kind of surprised that there were not more possibilities mentioned. Even 20 years ago, attacking Internet infrastructure seemed an obvious thing to do to us and we used to love talking about fun ways to ruin things over a burger at lunch. I mean is it really that hard to fathom people would think about attacking targets other than some organization, government, or other kind of company's servers?
> "But technology providers in the United States could suffer blowback. As Dyn fell under recurring attacks on Friday, Mr. York, the chief strategist, said such assaults were the reason so many companies are pushing at least parts of their infrastructure to cloud computing networks, to decentralize their systems and make them harder to attack."
Pushing your infrastructure to cloud computing is not decentralization - it's centralization, and we're all doing it. Imagine if an attack like this was against AWS... we'd all be screwed.
The downside of course, is that whilst their infrastructure can likely handle it, handling the bill associated with 'just scale up your service' could be worse than the attack itself.
Interestingly, the presenter notes that Amazon had seen a drop in DNS as an attack vector in 2015. I asked the presenter (Product Manager) why they hadn't productized the DDoS attack dashboard so you could be aware if you were being attacked (and it was being absorbed by AWS) and his response was that there was insufficient demand at that point to justify the developer staffing. He gave me his card and asked to request the feature so he could us it to make the case internally.
Does anyone here have stories of being successfully DDoS'd on AWS (other than by their own staff :) ?
Unless we can somehow secure every net-connected devices, ha (I don't know whether to cry or laugh right now)
Deleted Comment
If you want HA at local level you'd go with AWS AZs but if you need real HA you need can do the same at region-level.
Of course not everyone has the money/need to go down that route, but it's possible and even advised for some AWS services.
It decentralises that one company's DNS -- instead of having one or two DNS servers, perhaps at two sites, they now have 20, at 20 sites. If someone wants to target them, they're probably better protected.
But it's the same 20 servers as a million other companies, so the chance of those servers being a target is much greater.
Yeah, that's what I was getting at. I feel like my chances of being collateral damage on an attack against someone else is way higher in the cloud.
Even today with GitHub and other SaaS platforms going down, we were all affected.
But that's a fraction of the cloud. It's hard to integrate every service the hopeful equivalent of every other service.
I know of a company that pays an AWS bill sufficient to buy the equivalent of their pre-cloud datacenter's hardware every 1.5 months. The extra staff required to perform hardware maintenance would also cost about 2 months' worth of AWS each year (that means they're paying ~3x more than they would with hardware). Yet they moved to the cloud because it's the hip thing to do.
Cloud has upsides and things that are useful, especially for smaller proprietors who can take advantage of cheap droplets from DigitalOcean et al, but for grown-up companies, moving off your hardware shouldn't be automatic.
In that scenario you have a bunch of entrenched groups fighting about capex, capacity planning and budget all to get barely enough hardware to account for what you're doing in the next 3-12 months. Instead of taking a step back and creating a long term simple process for regular growth and replacement they get caught in the weeds because they have very old school mindsets.
Then you have your old school finance groups who are using terrifyingly delicate and complex interconnected spreadsheets to manage hardware expenditures and depreciation while maintaining old school draconian policies concerning CapEx budgets but allowing you to basically go nuts with OpEx.
You could try to change the culture in these entrenched groups who will view your attempts to make things better as political moves against them or you could just say "we're moving everything into the cloud" and make a complete end run around all of the people and baggage. The former is probably the "right" thing to do but the latter is going to let you focus on your product letting you get you back to being competitive.
This is only ironic if you expected moving to the cloud to be what provides the redundancy.
The BBC was affected by the Dyn outage not because they themselves relied on Dyn, but because components of their site did.
I fully agree with you about the paradox of how, in the intent to de-centralize we centralize into cloud VPSes and managed services.
The real reason for the move is that same showtune that we keep hearing in our heads and wish we could tune it out: it's cheaper to move from physical infrastructure to the cloud. It's cheaper to skimp on security by not updating IoT devices. It's cheaper to skimp on security because features need to come first. It's cheaper to outsource operational management to parties with less expertise in places that pay less. To spend less time securing infrastructure perimeters because it costs money.
We feel almost as if we feel comfort hiding behind heavyweights like Google and Amazon will protect us from the bad elements of the world, where we hear about major breaches every few weeks (eg., Yahoo being the most recent). Will this strategy pan out long-term?
With this DDOS, articles about machine learning picking up better password-cracking/guessing algorithms by having previously analyzed large volumes of passwords, major breaches in the financial world, talk of state-sponsored attacks (a la DNC emails) it certainly FEELS like the Internet has gotten a little bit more wild.
Consumer devices have to be more secure because if the low user skill level - and interest.
I am always reluctant to say "there should be a law against it" but frankly if we cannot mandate minimum standards of uogradbility and security for devices we will just keep handing over our devices to the first person to scan them.
A remote site shouldn't be able to get you banned from the Internet (by it's self); but it MUST be able to say, "This host is being abusive, restrain them from sending me data". ISPs SHOULD use that information to evaluate if a host from their network might be compromised or otherwise a negative player. ISPs SHOULD also take steps to inform, and link to educational resources, customers which are being bad citizens of the Internet. ISPs SHOULD also be financially motivated (punishments to them) for allowing too many uncivil customers online; this might take the form of instead banning that ISP from the Internet as a whole.
Okay, if I'm going to be liable, financially or otherwise, well, then we're gonna have to make some changes around here.
First off, I'm going to have to heavily filter and restrict what traffic you can send out to the Internet. What isn't filtered or restricted is going to have to be inspected, logged, and retained for a period of time.
Next, because I can't be certain that you're RFC3514 compliant and that at least some of the bits you're sending aren't malicious, I'm going to have to prevent you from sending out any encrypted traffic. Instead of allowing you to use any DNS servers you want, you're going to have to use mine (DNS is heavily abused for DDoS attacks). Outgoing e-mail will be automatically redirected to my internal smart host (STARTTLS will be blocked, by the way) and I'm gonna have to log, read, and retain it all. HTTP traffic will be transparently proxied and all requests and responses will be logged and retained.
That's just the beginning. Are you sure this is what you prefer as your "solution"?
As a network operator, I believe that your ISP should be nothing more than a dumb pipe and allow the bits that you send to pass through freely. As an ISP customer, that's how I want my ISP to act. (If something gets reported or I "notice" you for some reason then, sure, I'll look into it. Otherwise, I try to fuck with my customer's traffic as little as possible.)
I'll agree that there is certainly a problem, but it is not because of ISPs.
I agree with some of your points, but fracturing the internet is not a viable option. It might make sense if it were a healthy, competitive market instead of the near monopolies that exist today. Imagine banning Comcast, or AT&T.
The Internet has grown without proper planning using a lot of "quick and dirty" hacks (for example NATs, peering agreements) and today we just see the result. It reminds me of poorly designed email protocols that resulted in spam being the biggest part of email traffic.
If internet should wait until all use cases were created, it wouldn't exist. It's power was exactly that people could think on how to create things on top of was available. Many RFCs came afterwards.
The amount of consumer IoT currently connected with default and often outdated device settings is beyond belief.
Downside is that radio leakage licensing is fairly simple scientifically. Proving something is unhackable is harder ...
Deleted Comment
> Thirty-one states and the District of Columbia allow internet voting for overseas military and civilians. Alaska allows any Alaskan citizens to do so.
I had no idea any states allowed voting online. I wonder if the general population will ever get access to that.
Is this a reference I'm not getting, a speech-to-text error, or a simple misspelling of "absentee"?
There is a lot of talk of iot botnets but little to no evidence. This seems too vague and up in the air.
If all it takes is script kiddies and random extortionists to generate such large 1 Tbps scale attacks then we appear to be reliant on an unbelievably fragile base.
There is a growing realization of the need for more decentralization of services but these kind of attacks is going to drive more centralization if only Google scale companies can manage to stay up. I think this is drop everything and fix time for the IT profession.
"Mr. Assange is still alive and WikiLeaks is still publishing. We ask supporters to stop taking down the US internet. You proved your point. "
Link: https://twitter.com/wikileaks/status/789574436219449345
If their claim is true, does anyone think, it will turn many sympathizers against them? I don't think attacking normal bushiness is a good thing to do.
The motives of the attackers are much less interesting than the fact that such attacks are now possible.