'Nice Internet You've Got There… You Wouldn't Want Something To Happen To It…'
from the this-is-no-longer-theoretical dept
Last month, we wrote about Bruce Schneier’s warning that certain unknown parties were carefully testing ways to take down the internet. They were doing carefully configured DDoS attacks, testing core internet infrastructure, focusing on key DNS servers. And, of course, we’ve also been talking about the rise of truly massive DDoS attacks, thanks to poorly secured Internet of Things (IoT) devices, and ancient, unpatched bugs.
That all came to a head this morning when large chunks of the internet went down for about two hours, thanks to a massive DDoS attack targeting managed DNS provider Dyn. Most of the down sites are back (I’m still having trouble reaching Twitter), but it was pretty widespread, and lots of big name sites all went down. Just check out this screenshot from Downdetector showing the outages on a bunch of sites:
You’ll see not all of them have downtime (and the big ISPs, as always, show lots of complaints about downtimes), but a ton of those sites show a giant spike in downtime for a few hours.
So, once again, we’d like to point out that this is as problem that the internet community needs to start solving now. There’s been a theoretical threat for a while, but it’s no longer so theoretical. Yes, some people point out that this is a difficult thing to deal with. If you’re pointing people to websites, even if we were to move to a more distributed system, there are almost always some kinds of chokepoints, and those with malicious intent will always, eventually, target those chokepoints. But there has to be a better way — because if there isn’t, this kind of thing is going to become a lot worse.
Filed Under: attack, ddos, dns, internet, vulnerabilities
Companies: dyn
Comments on “'Nice Internet You've Got There… You Wouldn't Want Something To Happen To It…'”
Telling the infra-structure players alone must ‘do something’ is naive at best. The real culprits here are a mix of IoT and other hardware manufacturers that couldn’t care less about security. They need to be hurt for their lack of care where it hurts the most: their pockets.
So yes, the infra-structure portion can help mitigate the problem but unless we start taking security very seriously it won’t matter.
Of course, one must not forget the perpetrators should also be severely punished and if it’s a state actor maybe even cut it entirely from the network to preserve its health.
Re: Re:
It requires a number of things on the infrastructure side. Standard practice with IoT needs to be to have the devices on a separate non-Internet-connected network which requires the cooperation of router makers and users. Consumer routers need to implement RFC 3704 egress filtering by default. ISPs need to implement 3704 filtering on the customer side (the head-ends and/or CPE depending on physical configuration) and on the upstream side. Upstream networks need to implement 3704 filtering even if it means reconfiguring their topology to separate the non-transit parts of their network from the transit network. All parties involved need to stop depending on other parties to do the work and configure their own networks as if their measures are the only thing standing in the way of a massive DDoS attack. And finally, targeted parties need to be able to hold the originating and intermediate networks financially liable for all the costs involved, not just the small fraction of the access bill for the downtime, when those networks failed to enforce 3704 compliance.
That won’t stop all of it, but it’ll stop a huge portion of it. The rest can only really be dealt with by forcing end users (consumer or business) to clean up infected/compromised systems on their networks. Given the intransigence of the average end-user (whether a consumer or a company’s IT management) I don’t see anything short of big sticks wielded effectively having any effect.
Re: Re: Re:
Common typo, but means the opposite.
Re: Re: Re:
It also requires accountability, something we used to have on this network a few decades back, but no longer do.
The people whose infrastructure is responsible for this have to be held personally accountable. Publicly named. Publicly shamed, Publicly fired. Publicly denounced. Publicly humiliated.
Because it’s their fault. They’ve failed to meet minimum acceptable standards for Internet operations and they deserve to pay a steep price for it. Many of them should never work in this industry again.
Yes, that’s harsh, but having a big chunk of the Internet taken out — and the attackers could have done more and done it longer if they wishes — is a pretty big deal. Harsh penalties are appropriate.
And maybe, just maybe, everyone else will pay attention and start doing the things that they should have done 10-20 years ago in order to defend the Internet, not merely defend themselves.
Re: Re: Re: Re:
Okay, but who, specifically, are you referring to when you say “the people whose infrastructure is responsible for this”? Because TKnarr just named four different levels that need hardening (IoT manufacturers, router manufacturers, ISP’s, upstream networks).
Re: Re: Re:
“And finally, targeted parties need to be able to hold the originating and intermediate networks financially liable for all the costs involved, not just the small fraction of the access bill for the downtime, when those networks failed to enforce 3704 compliance.”
One point of contention, it’s probably minor to most. Say I order a private vlan from some IXP. Should the IXP be responsible for BCP38, after all the connection itself is just traversing their network to another provider. They certainly can not filter bogons, and how are they to know what ASNs or IPs should traverse that link.
Re: Re: Re:
I don’t know what 3704 is. Nor do I care.
on my net you will be stripped of IPV6.
any blocking rule should be in THREE unless you got a specific purpose
CUSTOM FORWARD
CUSTOM INPUT
CUSTOM OUPUT
ingress, egress, and forwarding
These devices getting hacked must be directly facing the web? Yes? I have several a SONY blue ray player right it has a 192.168.0.X I got a Marantz it has a 192.168.0.XX
Each IP needs rules to get out-crap works fine here and I got the youtube browser and the Opera browser in these boxes. All working just fine. Another thing is I constantly maintain a list of domain to IP’s so if DNS goes down I can load up techdirt at http://104.25.105.28 if i can punch thru cloudfare insanity.
People that don’t run their own boxes don’t get it. You can quote RFC’s all day long it’s freedom, tcpip and networking creativity that matter.
I seen a LOT of this wireless crap at the hospital, but is it even plugged in? I doubt it.
Re: Re:
“Of course, one must not forget the perpetrators should also be severely punished and if it’s a state actor maybe even cut it entirely from the network to preserve its health”
Assuming the identity of the bot-herder is known or can be discovered, It would be wise to shut down the botnet (not just the attack) prior to taking any steps to remove the herder or their network access.
If the botnet is reasonably intelligently designed, cutting the perp off from the internet may make it next to impossible to send a shutdown signal the C&C infrastructure will recognize.
not dyn, dyin
I think today they’re more like dyin dns. Sucks, though.
Started Again
It has clearly ramped up again and looks worse than it did.
Guardian website has been down for me for a while now. So, a suspected bomb in the Underground, a chemical attack on London City Airport, a very upset and crying Canadian trade minister, Russian aircraft carriers in the Channel and a massive internet attack.
What a day!
Re: Re:
Yeah right wrote:
Where did you hear it was an attack?
I haven’t seen anything (at least, anything from a reliable source) indicating they know the cause. Everything I’ve read so far says they’re still "looking into it".
Re: Re: Re:
well, yes that’s why I wrote suspected. No trace of any chemical has been found.
Was it a case of mass-hysteria or was it triggered?
Re: Re: Re: Re:
I’m going with mass hysteria as a far more likely explanation.
A fire alarm went off (accident or malfunction or someone being an idiot), someone smelled something (perfume, food, whatever), and then everyone panicked.
Re: Re: Re:2 Re:
I can see that. However, it also very easy to engineer. One person could do it. Set off an alarm, start coughing, maybe spray some perfume as you say. Someone being an idiot, or a calculated warning?
Re: Re: Re:3 Re:
I’m inclined to think people are perfectly capable of behaving foolishly without any help from nefarious outside forces. I’m also inclined to think that’s what happened here.
(Although the constant ZOMGTERRORISM encouraged by govt isn’t terribly helpful either.)
Re: Re: Re:4 Re:
Does London City Airport have a particular type of passenger on Friday evenings?
I agree it was probably a scary clown, but the timing isn’t foolish.
Re: Re: Re:5 Re:
Yes, City traders leaving the bars for their country retreats.
Re: Re: Re:5 Re:
Until or unless there is evidence to support this, all this speculation does is make people more paranoid and more like to panic at nothing.
And panicky people are dangerous. A panicked crowd is especially dangerous.
Re: Re: Re: No trace of any chemical has been found.
Not even any O₂? I wonder how the people there were breathing…
Re: Weeping Candian Trade Minister
She couldn’t get her own way; they insist on leaving ISDS in CETA and won’t give an inch on the choke points. Tough tizzy! Stay strong, Wallonia!
Anybody prepared to bet against this being used by governments and big business to restrict what the citizens can do, all in the name of stopping the bad guys.
Re: Re:
never waste a bad situation.
always use it to lie, cheat, and steal more liberty from the confused & ignorant plebs!
Re: Re:
Nah. Look at the sites affected. If you’re afraid of the citizens, you don’t cut off the bread and circuses.
Re: Re: Re:
It’s not bread and circuses it was a lot of sites people use to communicate with each other and share news, like Twitter and Reddit. With the internet down people can only get the news from the “government approved sources”. This site was also blocked for me for awhile, btw. Right before an election. Bet it happens again Nov 8.
Just have to share this gem of a quote from http://money.cnn.com/2016/10/21/technology/ddos-attack-popular-sites/index.html
(emphasis mine)
Re: Re:
That moment when you facepalm.
Re: Re:
OMG! Cyber activity in a DDoS?! Please God, no! Think of the children!!!
Re: Re:
Damn, why didn’t we think of cyber activity?! This whole thing could’ve been prevented.
Re: Re: Re:
We cybered too hard and now we’ve broken the cyber.
Re: Re: Re: Re:
That could be read a few ways…
Re: Re: Re:2 Re:
Hey now… What you do in the privacy of your own domain is your business.
Re: Re: Re:2 Re:
That could be read a few ways…
As a child of the 90’s, there is only one way to read it. I chuckle every time someone says “Do you cyber?” here, because that was exactly the same question folks said on BBS’s and the early internet back in the 90’s, but for entirely different, though very similar reasons.
Re: Re:
Quick .. call in the C.S.I. Cyber team… I’m sure they could watch for all the red dangerous code out there
“So, once again, we’d like to point out that this is as problem that the internet community needs to start solving now. … Yes, some people point out that this is a difficult thing to deal with. “
For a minute there I thought I was reading a quote about encryption from the FBI Director. Nerd Harder!
Nerd Harder!
I think Mike just suggested that somebody needs to nerd harder.
Re: Nerd Harder!
I think he’s actually suggesting people start giving a damn. It’s way below the nerd harder request.
Re: Re: Nerd Harder!
There’s an easy way to fix this.
Make companies financially liable for security issues in their products in a way that makes securing their software less expensive than not.
Until that happens, this type of issue isn’t going to get better.
Re: Re: Re: Nerd Harder!
how about jail time?
I am tired of the make people pay money bullshit. It just creates injustice.
People with money get to stomp all over others. The people harmed usually never get compensated while the government makes money off actual crime!
Re: Re: Re: Nerd Harder!
There’s an easy way to fix this.
Make companies financially liable for security issues in their products in a way that makes securing their software less expensive than not.
Sure, it’s just that easy if you think laws are vague, handwavy things.
In practice, what does this actually mean? Which companies are financially liable for security issues in which products? How quickly does the vulnerability have to be fixed to avoid liability? What’s the statute of limitations?
If there’s a vulnerability in the Linux kernel that affects Samsung phones, who’s liable? Samsung, Google, the Linux Foundation, all of the above? If the vuln has already been patched upstream, and Google’s already pushed an update, but Samsung isn’t staying up on Google’s updates, then presumably you’d hold Samsung liable but not Google or Linux, right? Okay. What if Samsung’s rolled the updates out on some phones but not others? What should Samsung’s obligation be for supporting its old phones? Should it be defined in terms of age? Userbase?
And you trust legislators to understand all these issues and write reasonable laws that take all of them into account while still being strong enough to discourage companies from releasing insecure devices?
You’re basically saying that legislators need to nerd harder, which isn’t really any better than saying programmers do. Though at least you had a suggestion for a way of fixing the problem, which is more than Masnick gave us in the article.
Re: Re: Re:2 Nerd Harder!
here’s a more solid start, based on use of MITRE’s CVE system.
Assume Samsung is selling IoT enabled toasters, because why not. Everything’s better with a network stack. Anyway, MSRP on this toaster is $100usd and Samsung releases the product Jan 1, 2017, and ships 1000 toasters.
Now, if there are no open CVE’s on any component of the IoT stack on this toaster in the 90 days before Samsung ships, they’re effectively insulated from liability. Oh, and in that world, the sky is Fuscia.
But, If there is an open CVE was announced >= 90 days before Samsung launches the product, and it gets exploited, Samsung is the hook for 5% of the MSRP for each unit sold of said product for every 90 days of age on the CVE.
Example: Samsung begins selling their IoT enabled toaster (MSRP == $100usd) on Jan. 1, 2017. And they sold 1000 of them on day 1. Said toaster has a vulnerability that was announced on Aug. 15, 2016 (just outside the 90 day grace period). If one of these toasters gets exploited and causes trouble, Samsung is going to write a check for (5% of $100) == $5 for each of the 1000 toasters sold as of the date of the CVE being exploited, plus the same fine going forward for each non-patched unit they sell.
Now, pretend that vuln wasn’t released on Aug. 1, 2016, it was release on Aug. 1, 2016. Same ship date, same quantity. Except now instead of 5% per toaster, it’s 10%. Add 5% for every 90 day interval of CVE age. Also, allow the total penalty per unit to exceed 100% of MSRP with no upper bound. So, you release an IoT enabled toaster with a 12 year old ssh vuln, and it gets exploited? assume qty 4-90 day periods / year to make it easy, now your penalty is (48 * $5) = $240 * 1000 = $240k in fines for each $100MSRP toaster you sold.
And why use MSRP as the basis for the penalty? Well, because it’s both easy to validate and publicly verifiable.
No grace period, no appeal, cut a check to a high school to fund a secure coding class, because CVE’s are public and theres no way the organization “couldn’t have known”.
Oh, and multiple CVE’s? 5% per CVE, and scale it out.
If you can verifiably patch these toasters 100% then you restart the clock from the time the patch was pushed to the toaster. If you can’t patch them, well, eventually you’ll get to write a check big enough to make the board pay attention.
Bonus: Specifically disallow said penalties as a loss for tax purposes.
As to your other question: It’s a Samsung toaster running a google code, Samsung pays. It’s their label. If Samsung wants to go back and fight it out with Google based on contract terms, that’s fine, Samsung can attempt to recoup their (already paid) losses from Google.
(yeah, I know. There’s no chance this or anything like it will ever happen.)
Re: Re: Re:3 Nerd Harder!
(ok, so that got long. Sorry about that).
But fundamentally, if we want anything resembling a secure IoT, we’re going to have to figure out a way to make it more expensive for companies to ship a vulnerable product than it is for them to fix it first, because the attack surface isn’t going to get smaller.
Re: Re: Re:3 Nerd Harder!
That’s a good and thorough answer, thanks.
Though it looks like there’s a typo:
Now, pretend that vuln wasn’t released on Aug. 1, 2016, it was release on Aug. 1, 2016.
Re: Re: Nerd Harder!
It’s a suggestion that the nerds need to do something, without any information whatsoever on what the nature of that “something” is. It is exactly “nerd harder”. It’s not quite as dire as the encryption backdoor debate (where the “nerd harder” advocates are pushing for things that are mathematically impossible), but it’s still not exactly helpful.
“Suggesting people start giving a damn” is vague to the point of uselessness too. Which people? “The internet community”, apparently. Whatever the fuck that means.
So, once again, we’d like to point out that this is as problem that the internet community needs to start solving now.
May I point out to Techdirt that we are (see Hyperboria: http://hyperboria.net/ for an example), but that there is serious difficultieswith deploying any such technology. The vast majourity of people (corporate & individuals) can’t be bothered upgrading (most of whom won’t see the point), and many who can be bothered won’t do so as it (if not engineered correctly) will risk backwards incompatibility.
Engineering around these difficulties is a significant challenge I’ve only seen begin to be solved solved recently (and hyperboria could still be improved here).
Tl;dr Don’t ask us to start solving the problem: we have. Instead do what little you can to help us deploy it.
Re: Response to: Anonymous Coward on Oct 21st, 2016 @ 11:57am
Sorry for the bad formatting, commenting from my phone.
The first paragraph is a quote from the artical.
Nerding harder...
The fundamentals are that I can’t *trust* my own computer, let alone yours.
Lacking trust in computers, *everything* is going to have to go to a bit-torrent style model with no central host (somebody already did this for websites, I forget the project name) because there are enough broadband IoT devices out there to DDOS any single individual, company, or any device performing a particular function. The biological analog should be obvious.
And, just as with fair use and copyright, the problem of discerning “legitimate” traffic (all of Techdirt’s fans) from “illegitimate” traffic (all of Techdirt’s haters, and 100 million of their bots, coordinated so they look just like its fans) is basically impossible.
Time to break the glass over the emergency tools and prepare for the internet to go down. Probably November 9.
Re: Nerding harder...
If you want distributed DNS: http://dot-bit.org/Main_Page
For the websites, https://webtorrent.io/
Re: Re: Nerding harder...
Sorry, webtorrent.io was just the WebRTC part of Web2Web, which is down right now: https://github.com/elendirx/web2web
Re: Re: Re: Nerding harder...
Throw in another link as an alternative to Webtorrent/Web2Web:
http://ipfs.io/
Re: Re: Re:2 Nerding harder...
Decentralized distributed file sharing?
aMule with Kademlia
http://www.amule.org
If FaceTwit isn't available . . .
If FaceTwit isn’t available, then a certain presidential candidate will be unhappy. I won’t name any names. But he or she likes to sit on his/her solid gold toilet bowl at 3 AM using FaceTwit.
A service outage could be a reason to push the big red button.
Re: If FaceTwit isn't available . . .
Tweedledum and Tweedledee
Agreed to have a battle;
Nerd Harder
“So, once again, we’d like to point out that this is as problem that the internet community needs to start solving now…But there has to be a better way.”
Hey, isn’t this YOU saying “Nerd harder!”?
I get it, this problem isn’t intractable, but still…
Re: Nerd Harder
Honestly, it isn’t nerd harder. For IoT, if the developers are too lazy to patch vulnerabilities than simply use a distro that will and setup a cron job to check and update automatically. For network operators, the BCP38 guidelines and BGP filtering will greatly reduce the possibility of your customers doing this from your network.
IE The tools are there, people just are not using them.
Re: Re: Nerd Harder
So create an attack vector, the update server.
Not to mention the central repository it creates of users of that device/software for targeted attacks.
Re: Re: Re: Nerd Harder
Tho I don’t know if there is any good solution to this problem. Have regular patching, thus introducing an attack (and privacy) vector, or don’t patch, thus avoiding that vector, but leaving yourself open to pre-existing flaws in the code…
This isn’t getting nearly enough coverage as it should. I managed to catch an article on yahoo news (yeah yeah, laugh it up) about “temporary” 2 hour outages for some people on the east coast.
However, I cannot access the websites of some pretty major companies, such as soundcloud and twitter. If I used twitter, that might be an issue for me. But I know that a lot of people rely on it for their breaking news, and with a lot of other big name company sites down we cannot get up-to-date info.
This is scary bad. The fact that Amazon’s web service went down is scary. Big companies rely on AWS for their internet connectivity for things, and if that goes/stays down, it can mean a lot of lost income.
Re: Re:
“This isn’t getting nearly enough coverage as it should”
Probably because it isn’t hitting everybody. If I wasn’t reading about it on the news sites I’d never have known. Been online in CST since before 6am, have used many of the major sites mentioned (and of course AWS at the back of many) all morning with no indication of any problems. (I don’t use FB but I have been using Amzn, TWTR, NYT, WAPO etc etc etc, major sites for work, and they’ve all been flying. Weird.) Literally except for reading about it I have not noticed anything. I feel left out.
Re: Response to: Nick on Oct 21st, 2016 @ 1:36pm
You have never heard of “Frontier” as in the company that just purchased Verizon’s FIos while they were rated 270 out of 278 different customer service providing entities. What good is these government bodies created to help consumers from being ripped off when a company (with nearly the worst CS rating) that has some money can purchase Verizon’s Fios service when Verizon was the internet providers leader in customer service. How the hell is that protecting the consumers.
Yes, it’s bad for Amazon but what about other small businesses that are totally revenue-dependent in their internet services staying up. There were companies in Florida with no internet service for a month and many more for weeks. Frontiers tech’s didn’t show up for appointments and when CS was contacted they just lied. One idiot called the consumer in the same landline he was there to repair to let them know he was there. They provided their cell phone numbers no less than 7 times for these brain-dead idiots. Mean-while they were chastising Warner Cable for over charging and throttling only to implement the exact same pricing structure except worse.
WTH!
If only the internet had been envisioned as a distributed system, resistant to single-points of failure. /s
Re: Re:
Laughs… What amazed me was EBay had redundancy built into their DNS, but PayPal didn’t. I guess the right hand really doesn’t know what the left hand is doing.
Re: Re: Re:
Paypal was spun off from eBay over a year ago.
Easier to blame Russia for it since they don’t have any other reason to make up for trying to start another war.
Fix it: White Hat Hacking
Start scanning and when you find a device with a default password, sign in and change it to something random.
If they can find them, so can we. And if the user can’t get in, they will just reset it to default. And it will be found again. Repeat.
Have done this dozens of times in the large and small companies I’ve worked for. Camera’s, scanners, printers, et cetera. If the customer/employee calls in a tech support ticket, they are talked thru how to reset, configure and set a good password.
Secondly, maybe some enterprising company/person could set-up a simple “Certified Safe Supported”. A small company could get a product, certifiy that it has security in ind, such as a) support for updates b) obvious passwords are not used/repeated c) I really don’t need to list them…
Desperately need MaidSafe's SafeNetwork to stop this nonsense
SafeNetwork would have prevented all these DDoS attacks, it is time to have a fully distributed internet for once and for all.
Where are the IoT apologists...
that used to hung out here claiming the IoT industry shouldn’t be held responsible because it’s so “innovative”? They seem to be strangely quite right now.
Heads Will Roll
…once someone points out how badly this sort of action can impact the Zetas’ online scamming “business.”
When you outsource to the cloud, you have a SPOF you can't see.
Whether or not Dyn should have been able to withstand this DDOS, whether or not the DDOS should have been prevented, it’s still a problem for all of Dyn’s customers that decided that they didn’t need any other DNS services because Dyn is the cloud.
On the DNS customer side, there’s no reason not to use multiple authoritative DNS providers, including running one yourself. The cleanest way of doing this is to run two or three widely separated DNS servers that only talk to your three DNS services. Even for huge zones, this is a cheap and idiot-resistant method.
On the resolving side, there’s no excuse for not having two or three nameservers listed on each of your computers. If you are small: one from your ISP, one from Google, one from any other service. If you are in any position to run caching DNS servers, do that as well.
Taking Credit
http://www.sfgate.com/business/technology/article/US-internet-disrupted-as-key-firm-gets-hit-by-10027979.php
I do love the title
and cue “it’s terrorists / encryption” to blame so we need to take away your civil liberties / destroy the constitution in 5.4.3.2….