This is a pretty thorough and high profile hack on a major tech company - this isn't something I'd expect from an Amazon owned property. The hack (allegedly, I haven't downloaded it) includes
* Entire git histories
* Internal/Private AWS SDKs
* Encrypted Password dumps and payout reports
It's so comprehensive I'm very curious into how an attacker got that level of access. I can't think of another, large, corporate web 2.0 startup who's gotten owned in a similar fashion. Could the same attack work on Amazon? YouTube?
It's also strange that someone who has this level of access to what is presumably a multi-billion dollar company decided to just leak the data? Maybe they did try to ransom it, but I'd imagine someone with this kind of access inside Twitch must have had some creative way of making money.
There were no encrypted password dumps. No production secrets were leaked (according to the article). What's here is no more than what your average Twitch engineer has access to.
Yes, that included payout data. Anyone with "staff" access to the site (which any employee can have) has access to any streamer's dashboard, which includes payout data.
I don't think this was an attack. Based on the data so far I think it was a disgruntled engineer. Obviously if more gets leaked later I may revise that opinion.
I also worked for Twitch and can confirm what you're saying is true. These repo's any staff member had access to - including non-engineering staff.
Revenue for the longest time was as simple as navigating to a streamers dashboard as staff, but they did finally gate that away from staff who don't need to see that info, however I am sure there are other ways to obtain revenue reporting info.
I am assuming all data - including personal - has been compromised but so far, the data leaked is data that most staff would have access to in some way or another. Some may find that shocking, but this was not a "high level hack"
No one in IT should have access to business data. That's simply best practice. Worst case would be a database engineer who has access to backups or some prod data for troubleshooting, and even that should be under tight control with good access accounting.
> It's also strange that someone who has this level of access to what is presumably a multi-billion dollar company decided to just leak the data? Maybe they did try to ransom it, but I'd imagine someone with this kind of access inside Twitch must have had some creative way of making money.
Notably, the initial leak didn't actually include the password data which the leaker claims to have, just source code and payment data which has been verified by several affected streamers. It's possible that this first leak was just to establish trust so they can random or auction password hashes later.
Given the torrent is labeled "twitch-leaks-part-one" I'm curious too as to what they have. The torrent breaks out into a lot of compressed volumes, so it's clear this wasn't just a backup file, but a curated collection of files. I'm very curious if we will see any other amazon related leaks come from it.
Either way, I can only imagine the chaos inside as they try to figure out what has transpired here.
>It's possible that this first leak was just to establish trust so they can random or auction password hashes later.
Password hashes are relatively useless though? Once the leak is announced I imagine most of the big targets will rotate their credentials. Then the next thing you need to do is spend possibly thousands in CPU time bruteforcing bcrypt hashes. Then I'm not sure what you can even do with those.
I'm not criminally creative but I imagine you could make more by abusing trust with payment processors or fraudulent invoices.
Maybe that Twitch is competent in the password department so they decided against it? But thinking about it, although it's unclear if two-factor secrets are included in the leak, but maybe the two-factor secrets may be usable to someone who has already the password of a victim. Unless it's the dongle-type one (WebAuthn/FIDO), the secret is common to both the server and the user, so two-factor bypass is almost certain in this case.
Doesn't seem likely to me. If the attacker has password hashes then they would want to keep this attack quiet so that the buyer of the hashes would have time to compute the passwords. If Twitch gets wind of this happening then a simple password reset would foil any efforts.
I'm hoping we will get to see a transparent report (from hacker or Twitch) on how this happened.
I think anyone would be excited to hack Twitch as the site alone - or any big platform for that matter - but this is quite literally someone just downloading the entire Twitch ecosystem and publishing it online.
It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.
Gosh - I've worked at shops where we handled multi-terabyte images and we'd regularly stream large chunks of that while debugging tools. I've also worked at places where data was king and 125GB of stuff might be a reasonable dispatch of data to help someone debug.
The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs.
I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.
Trying to protect against leaking developers/employees is like trying to protect against lone gunman terrorists: useless.
And, if you try anyway, it is likely to cause more annoyance to everyone involved than actual protection (think TSA).
There was a fad for tools that accomplished this in enterprise networks, with much clearer rules for who needs to access what (it was called "data loss prevention", or DLP) and those tools for the most part don't work. This is a harder problem than it looks like.
> It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.
Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere.
Plus, you don't have to exfiltrate 125GB in one go.
I feel like once you have it pulled downm, it would be as simple as an upload to s3 (which wouldn't trigger any flags), then making the bucket public whenever you want. Hell, S3 used to (still does?) support being part of a torrent swarm...
There are so many indiscreet USB pentesting devices easily purchasable by anyone today, I'm actually surprised this sort of thing doesn't happen more often.
ITT: people shocked that something like this could happen at a company the size and profile of Twitch.
Running security at scale in a hypergrowth B2C company is very difficult. It's also completely different from running security at a startup, in a B2B company, or a slower-growth situation. _Every_ security executive and manager I've met has given up in frustration after 12-24 months and gone to take a cushy FAANG job instead.
I'm not surprised at all. My experience in security at a larger SV unicorn was that changes only happened in the immediate aftermath of a security crisis. Otherwise, there was incredible inertia and you just wouldn't be able to get the institutional support you needed to make progress.
How much of this is a holdover of lax security practices from before they were acquired? I can’t imagine AWS being managed in a way where local network access gives you keys to the kingdom. Then again, EC2 instance profiles do let you do quite a bit.
I always had the impression that Twitch were operating in a largely independent fashion. For instance, it had been an open secret for years that one of their executives had been sexually harassing female streamers. Only a year ago he was finally fired. If Amazon had a firmer grip on Twitch, I'm sure they would have stepped in much earlier.
If you go back to the Adobe software breach circa 2013, a large part of their issues were the bolt on connections between acquisitions. It's honestly the most common thing I see in the startup world.
> It's also strange that someone who has this level of access to what is presumably a multi-billion dollar company decided to just leak the data?
From what I heard about Twitch-interns over the years, it seems the company is more a third-rate-s**hole that grew too big too fast and accumulated a huge amount of technical debt and fatal security flaws. Making billions doesn't mean anything if you don't invest them back into the important corners of the company. It's considered a miracle that the platform is still working that well in that state. And what comes from the leaks so far supports this view.
Though, said that, it seems they did start to improve one or two years ago, just too late to prevent this critical hit. But considering this was also a strike that avoided the deadly parts (yet), maybe there is a different aim here and the company can grow from this? It will be interesting to see how Amazon will react to this.
> From what I heard about Twitch-interns over the years, it seems the company is more a third-rate-s*hole that grew too big too fast and accumulated a huge amount of technical debt and fatal security flaws.
I mean this as a genuine question, but is there any company that didn't end up like this after an exponential growth phase? I'm not saying it's okay, but this feels par for the course. I've now been at two start ups during that hockey stick growth time and both went through this as well.
I'd be curious if anyone here has worked at a large, fast growing tech company where they didn't accumulate a ton of technical debt during growth. If so, what did the company do to prevent that?
Does anyone know if Twitch employees have two factor auth? Having access to an employee's account would be the easiest way to pull this off.
It'd be strange if they don't have two factor auth, of course, but it's just as strange to have this large of a hack.
I think if it is a simple case of an employee account takeover, then the attack would "work" to some extent at any company. Larger companies typically have strict data access requirements, though. Good luck finding the few employees who have raw access to Google password hashes, for example. And even more luck knowing how to get that data if you do.
Every Twitch Developer has 2FA even 3rd party developers are required to have 2FA I also think, but don't know, that this applies to Twitch Broadcaster Partners as well in order to have their tax information in the system.
Luckily iirc from a conversation with a senior Twitch engineer the Tax information backend has been migrated to Amazon. So hopefully that did not leak... Because that would be full legal name and addresses of a ton of streamers that likely have stalkers.
From an ethical standpoint, any code that amplifies and profits from radical speech should be fair game for release. If employees or hackers feel the need to release info in that regard, so be it. This is the risk defined in such models and should be mitigated accordingly.
>Because you expect Amazon to put security priority over new features and profit?
I don't know what you think Amazon stands for, but Amazon runs the largest cloud hosting service in the world - AWS, which not only runs a large number of other large companies but governments as well. I know, first hand, that their datacenter security protocols are state of the art.
Amazon has a much larger surface attack area so if they were playing fast and loose with security, chances are we would know already.
EC2, Amazon's cash cow, competes with nearly identical offerings from Microsoft and Google, and is not a place where additional features are often all that valuable to customers. Any sort of breach like this on EC2 would seriously hurt Amazon's bottom line and they know it.
It is really fun to go through the source code. You'll find interesting architecture diagrams, documentation etc. It's like joining a new job and being amazed how a service you actually use was build.
I would love to see someone look deep into Twitch recommendation system - last time I tested the thing they call "Feedback" is a rolling buffer and wont let you exclude more than ~100 things, adding more simply removed oldest entries and started spamming you with things you already excluded in the past. This looked like performance optimization (less things to track per user).
This won't help with preroll ads because the video segments themselves are replaced in the stream data. They're not ads, but it's not the stream either.
You get a "twitch commercial break in progress" video for the time the ads are playing.
I watch all of my twitch using mplayer. "magic incantations" when generating access token is what produces ad free .m3u8. For example early methods involved setting origin and/or referrer headers to internal Amazon systems.
I'd be interested if someone could get their own instance of Twitch up and running from this leak. Someone mentioned internal API's, which would have to be reworked to avoid detection, but it'd be interesting to host it on AWS just to see how long it takes to get shut down.
How would current AWS policies hold up? Obviously the code would be illegally acquired, but do they have detection mechanisms in place?
Even with source code it is hard to run a service if not impossible. You would need well written documentation that explains various options and error codes you could potentially get.
Many times there is some magic command only one guy knows and he will share with you on slack.
Rubbing a service of any complexity takes years of institutional knowledge.
100s of services and databases to work out and sort through. Good luck building a global real-time video CDN too. You could build your own faster. Microservice architectures mirror the org that built them. You wouldn’t do it the same way for yourself.
Readit News
* Entire git histories
* Internal/Private AWS SDKs
* Encrypted Password dumps and payout reports
It's so comprehensive I'm very curious into how an attacker got that level of access. I can't think of another, large, corporate web 2.0 startup who's gotten owned in a similar fashion. Could the same attack work on Amazon? YouTube?
It's also strange that someone who has this level of access to what is presumably a multi-billion dollar company decided to just leak the data? Maybe they did try to ransom it, but I'd imagine someone with this kind of access inside Twitch must have had some creative way of making money.
Yes, that included payout data. Anyone with "staff" access to the site (which any employee can have) has access to any streamer's dashboard, which includes payout data.
I don't think this was an attack. Based on the data so far I think it was a disgruntled engineer. Obviously if more gets leaked later I may revise that opinion.
Revenue for the longest time was as simple as navigating to a streamers dashboard as staff, but they did finally gate that away from staff who don't need to see that info, however I am sure there are other ways to obtain revenue reporting info.
I am assuming all data - including personal - has been compromised but so far, the data leaked is data that most staff would have access to in some way or another. Some may find that shocking, but this was not a "high level hack"
Saying that no 'secrets' were leaked is effectively burying the lede.
Deleted Comment
Notably, the initial leak didn't actually include the password data which the leaker claims to have, just source code and payment data which has been verified by several affected streamers. It's possible that this first leak was just to establish trust so they can random or auction password hashes later.
Either way, I can only imagine the chaos inside as they try to figure out what has transpired here.
Password hashes are relatively useless though? Once the leak is announced I imagine most of the big targets will rotate their credentials. Then the next thing you need to do is spend possibly thousands in CPU time bruteforcing bcrypt hashes. Then I'm not sure what you can even do with those.
I'm not criminally creative but I imagine you could make more by abusing trust with payment processors or fraudulent invoices.
I think anyone would be excited to hack Twitch as the site alone - or any big platform for that matter - but this is quite literally someone just downloading the entire Twitch ecosystem and publishing it online.
The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs.
I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.
I am trying to recall, but I am pretty sure when I worked in Microsoft Office that a build would pull down many tens of gigabytes of data.
125GB in one day from the build system wouldn't be uncommon!
Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere.
Plus, you don't have to exfiltrate 125GB in one go.
Deleted Comment
So let's say someone with access to all GitHub repos gave the password to someone else, maybe then it was downloaded from another machine?
Or someone stole the credentials and downloaded from another machine?
Or someone got access to such a machine?
It's it not possible to prevent these cases?
How long does such a download take?
Running security at scale in a hypergrowth B2C company is very difficult. It's also completely different from running security at a startup, in a B2B company, or a slower-growth situation. _Every_ security executive and manager I've met has given up in frustration after 12-24 months and gone to take a cushy FAANG job instead.
I'm not surprised at all. My experience in security at a larger SV unicorn was that changes only happened in the immediate aftermath of a security crisis. Otherwise, there was incredible inertia and you just wouldn't be able to get the institutional support you needed to make progress.
Within Amazon those are almost going to be two entirely separate companies, with very different security focuses.
The idea that Amazon is monolithic and uniform wasn't true when I left there in 2006, and I'm certain it is less so now.
And that isn't just that its related to the merger, but that fundamentally its different business orgs with different focus.
From what I heard about Twitch-interns over the years, it seems the company is more a third-rate-s**hole that grew too big too fast and accumulated a huge amount of technical debt and fatal security flaws. Making billions doesn't mean anything if you don't invest them back into the important corners of the company. It's considered a miracle that the platform is still working that well in that state. And what comes from the leaks so far supports this view.
Though, said that, it seems they did start to improve one or two years ago, just too late to prevent this critical hit. But considering this was also a strike that avoided the deadly parts (yet), maybe there is a different aim here and the company can grow from this? It will be interesting to see how Amazon will react to this.
I mean this as a genuine question, but is there any company that didn't end up like this after an exponential growth phase? I'm not saying it's okay, but this feels par for the course. I've now been at two start ups during that hockey stick growth time and both went through this as well.
I'd be curious if anyone here has worked at a large, fast growing tech company where they didn't accumulate a ton of technical debt during growth. If so, what did the company do to prevent that?
It'd be strange if they don't have two factor auth, of course, but it's just as strange to have this large of a hack.
I think if it is a simple case of an employee account takeover, then the attack would "work" to some extent at any company. Larger companies typically have strict data access requirements, though. Good luck finding the few employees who have raw access to Google password hashes, for example. And even more luck knowing how to get that data if you do.
Yes, IIRC everyone at Amazon has a hardware security key (which is more secure than the standard mobile app TOTP most of us use everywhere online).
Luckily iirc from a conversation with a senior Twitch engineer the Tax information backend has been migrated to Amazon. So hopefully that did not leak... Because that would be full legal name and addresses of a ton of streamers that likely have stalkers.
https://www.theguardian.com/technology/2012/feb/17/facebook-...
…except Mangham didn’t ever get to release his spoils to The Internet?
Linkedin, Microsoft, Yahoo, Google
Dead Comment
And if speech is "radical" meaning to the point of illegality, shouldn't the legal system decide, rather than the court of public opinion?
Because you expect Amazon to put security priority over new features and profit? We have very different understandings of what Amazon stands for.
I don't know what you think Amazon stands for, but Amazon runs the largest cloud hosting service in the world - AWS, which not only runs a large number of other large companies but governments as well. I know, first hand, that their datacenter security protocols are state of the art.
Amazon has a much larger surface attack area so if they were playing fast and loose with security, chances are we would know already.
https://www.twitch.tv/deepfrieddev
Too bad, it would be nice to see someone go through and document how Twitch works. I've never worked at "web scale" so I'd probably learn a lot.
As someone who has worked at both large and small companies, you'd probably be disappointed.
Everyone interested, just download the code :)
The chat had a few Amazon insiders, which was interesting to read their perspectives.
Deleted Comment
Dead Comment
The download was posted to 4chan today, described by its unidentified source as “part one” of “an extremely poggers leak,”
> Calling Twitch a “disgusting toxic cesspool,”
This will help with ad preroll blockers.
I would love to see someone look deep into Twitch recommendation system - last time I tested the thing they call "Feedback" is a rolling buffer and wont let you exclude more than ~100 things, adding more simply removed oldest entries and started spamming you with things you already excluded in the past. This looked like performance optimization (less things to track per user).
You get a "twitch commercial break in progress" video for the time the ads are playing.
You can check this by loading a stream with MPV.
>You can check this by loading a stream with MPV
I watch all of my twitch using mplayer. "magic incantations" when generating access token is what produces ad free .m3u8. For example early methods involved setting origin and/or referrer headers to internal Amazon systems.
How would current AWS policies hold up? Obviously the code would be illegally acquired, but do they have detection mechanisms in place?
Many times there is some magic command only one guy knows and he will share with you on slack.
Rubbing a service of any complexity takes years of institutional knowledge.
https://news.ycombinator.com/item?id=28770590