Apple Developer Website Update

Here's my semi-educated guess for how the attack started: from casual observation (view source, URLs ending with .action, etc) a good chunk of the ADC is written in Java and uses WebWork/Struts2, a framework I helped create years ago.

Late last week a security advisory came out that allows for executing malicious code[1]. Atlassian, which uses similar technology, also issued announcements around the same time[2]. My wild speculation is this was the attack vector.

Sadly, I feel some responsibility for this pretty major security hole. There have been a few like this and they are all rooted in the fact that almost 9 years ago I made the (bad) decision to use OGNL as WebWork's expression language. I did so because it was "powerful" but it opened up all sorts of extra binding trickery I never intended. I haven't been contributing to the project in 5+ years, but this is a good reminder how technology choices tend to stick around a lot longer than you ever imagine :)

[1] http://struts.apache.org/release/2.3.x/docs/s2-016.html [2] https://confluence.atlassian.com/display/BAMBOO/Bamboo+Secur...

colanderman · 12 years ago

technology choices tend to stick around a lot longer than you ever imagine :)

It amazes me how true this is. I've learned that assertions such as "this is a mockup and should be replaced ASAP for reasons X Y Z" tend to get ignored by inheritors of proofs-of-concepts for as long as (or longer than) possible. My coworkers wonder why now I fight tooth-and-nail to (from their perspective) over-engineer things from the start; I know that short-sighted decisions will never be revisited until it's too late.

karlkatzke · 12 years ago

This is the biggest reason that I hate all of the "go fast and break things" culture around here. There has to be a balance, but big names and investors don't seem to be encouraging them.

joeblau · 12 years ago

So what you're saying is that it's your fault that I can't update my provisioning profile? No just kidding--Honestly, while you may feel responsible, the project has been up and running for a while. The current maintainers are responsible for the bugs that show up regardless of who designed the original code base. Definitely appreciate you shedding light on the situation since Apple hasn't really given us to much information.

aboodman · 12 years ago

Both are responsible, and responsibility can add up to more than 100%.

czhiddy · 12 years ago

The timing of this looks right: Struts 2.3.15.1 was released on 7/16/13, and developer.apple.com went down on 7/18/13 for "maintenance". Plenty of time for an enterprising black hat to notice and exploit the vulnerability.

> Sensitive personal information was encrypted and cannot be accessed, however, we have not been able to rule out the possibility that some developers’ names, mailing addresses, and/or email addresses may have been accessed.

So they can't rule out the possibility that sensitive personal information, which cannot be accessed, has been accessed. Got it.

Apparently our intelligence, which cannot be insulted, has been insulted.

kristofferR · 12 years ago

By "sensitive personal information" they probably just mean passwords and credit card information, not names, email addresses and mailing addresses.

hdivider · 12 years ago

What I find slightly unnerving is that Apple didn't make this clearer.

If they know that credit card information was not affected, they should say that. E.g. "Sensitive personal information (such as credit card data) was encrypted and cannot be accessed, ..."

It's reasonable to suppose that 'sensitive' includes credit card information, but as it stands it's something we have to interpret.

I'd suggest we all check our credit/debit card statements more often over the coming days, just to be sure. =)

_delirium · 12 years ago

Passwords could be hashed, but credit-cards are the big one you have to keep in plaintext. If you want to bill the card without asking for the number to be reentered, there's no way to avoid storing the number and expiration date. PCI does mandate that you keep less than necessary to initiate a new charge, though: you are not allowed to store the 3-digit verification code from the back of the card. Future charges from the same vendor can go through based on the stored information (without re-sending the verification code), but charges from a new vendor would need the code, so this is intended to make it harder for someone who stole the saved information to initiate a new charge. A loophole is that in-person charges do not use the verification code, so someone could use the saved information to fabricate physical cards, and try to use them at stores (the U.S. doesn't typically use either chipped or PIN-protected credit cards, so cloning a card from the number is relatively easy, prevented more or less only by the heuristic fraud-detection algorithms).

smegel · 12 years ago

I think I would rather someone have my CC number than my home address (which would be the same as my mailing address).

bennyg · 12 years ago

I'm imagining bank account numbers over CC info/passwords was the sensitive part.

llamataboot · 12 years ago

"First of all, this does not affect iTunes customer accounts—this is a different system and all iTunes customer information is completely safe, Apple told me.

It’s also important to note that the hacker did not get access to any app code or even the servers where the app information was stored. The hacker also did not get access to any credit card information.

The only thing that the hacker could have gotten access to was the names, email addresses and mailing addresses of the developers. At this point, Apple doesn’t know if the hacker even managed to see that information. Worst case, that is all the information they would have seen, according to Apple."

http://www.loopinsight.com/2013/07/21/apple-comments-on-deve...

jordanthoms · 12 years ago

Alternately, inside the reality distortion field developers’ names, mailing addresses, and/or email addresses is not sensitive personal information.

Turing_Machine · 12 years ago

Anyone who has my name can find my email address with a simple Google search. My mailing address is on all kinds of public records.

ojbyrne · 12 years ago

If by "reality distortion field," you mean the legal community, then yes. Having been through something similar recently, the lawyers will likely be making sure everyone involved understands the concept of PII: https://en.wikipedia.org/wiki/Personally_identifiable_inform...

ptwiggens · 12 years ago

How are names, mailing addresses, and email addresses sensitive personal information?

I would imagine that for most of the people signed up, it wouldn't be that hard to track down their name and email just from knowing the name of their app.

VeryVito · 12 years ago

One of the understood requirements of publishing an app on the App Store is that developers must provide some means for customers to contact them directly (support page, email address, etc). If you're selling apps on the app store, people can already peddle their wares to your email account.

So yeah, developer's names, addresses and emails are not secrets by any means. Why would anyone buy an app from someone they had no means of identifying?

clarky07 · 12 years ago

names, email adresses, and mailing addresses aren't particularly sensitive. These are all pretty easy to get for most people without hacking anything.

jpttsn · 12 years ago

Apple apparently doesn't agree that, in context of the data they have on you, your name, mailing address or email address qualifies as sensitive.

Deleted Comment

_sabe_ · 12 years ago

"The intruder had good intent with trying to "secure" our personal information. But despite nothing being hacked, as it was only a 'threat', we still need to tear down and build up the system from scratch again. In the spirit of transparency we've waited 72 hours before giving you this nonsense bullshit. Please note that some (that is all) of you will from now on get regular viagra offerings in cyrillic. Good for you!"

rimantas · 12 years ago

  secure
  verb [ with obj. ]
  
  2 succeed in obtaining (something), especially with difficulty

Deleted Comment

grep -E "Revocation Date: Jul 17 .{8} 2013" wwdrccrl.txt | wc -l 3065 grep -E "Revocation Date: Jul 18 .{8} 2013" wwdrccrl.txt | wc -l 2289 grep -E "Revocation Date: Jul 19 .{8} 2013" wwdrccrl.txt | wc -l 2 grep -E "Revocation Date: Jul 20 .{8} 2013" wwdrccrl.txt | wc -l 0 grep -E "Revocation Date: Jul 21 .{8} 2013" wwdrccrl.txt | wc -l 0

grep -A 3 -B 1 -E "Revocation Date: Jul 19 .{8} 2013" wwdrccrl.txt Serial Number: 2628C7F90970D227 Revocation Date: Jul 19 03:14:04 2013 GMT CRL entry extensions: X509v3 CRL Reason Code: Key Compromise -- Serial Number: 1A51ABFA4844BD45 Revocation Date: Jul 19 03:24:03 2013 GMT CRL entry extensions: X509v3 CRL Reason Code: Key Compromise

Lightbody · 12 years ago

jpdoctor · 12 years ago

tcas · 12 years ago

I downloaded the CRL for developer certificates [1] and quickly looked at it using grep:

These are the two certificates that were revoked on the 19th

To generate the wwdrccrl.txt file I used:

  openssl crl -inform DER -text -noout -in wwdrca.crl > wwdrccrl.txt

Just to be clear -- every entry there I see lists the reason as Key Compromise, just interesting that they usually seem to revoke at least 2000 certificates a day but suddenly stopped on the 19th with just revoking 2.

[1]http://www.apple.com/certificateauthority/

cmelbye · 12 years ago

That's because the portal is down and people can no longer log in to revoke their developer certificates.

That's what I thought -- but it was also down the 19th yet 2 were revoked.

Probably means nothing however. I doubt that anybody with the ability to get into the system would want to get only developer certificates.

dakrisht · 12 years ago

"Completely overhauling our developer systems, updating our server software, and rebuilding our entire database."

That does not sound like an intruder "attempt" by any means.

They got hacked, and they got hacked bad if they're rebuilding databases and overhauling entire enterprise-class systems over there.

Transparent my ass. They're deep in the gutter, 3-days and counting no fix, engineers are probably working 24 hours a day and the entire site is still down. This isn't a small time breach folks. They had to go public considering it will probably be down for a few more days...

sarreph · 12 years ago

A little more info from TC: http://techcrunch.com/2013/07/21/apple-confirms-that-the-dev...

Update — Just got off the phone with an Apple rep, who confirmed a bit more:

- The hack only affected developer accounts; standard iTunes accounts were not compromised

- Credit card data was not compromised

- They waited three days to alert developers because they were trying to figure out exactly what data was exposed

- There is no time table yet for when the Dev Center will return

johansch · 12 years ago

There is an interesting comment at techcrunch:

http://fyre.it/tjlVmC.4

"[...] One of those bugs have provided me access to users details etc. I immediately reported this to Apple. I have taken 73 users details (all apple inc workers only) and prove them as an example.

4 hours later from my final report Apple developer portal gas closed down and you know it still is. I have emailed and asked if I am putting them in any difficulty so that I can give a break to my research. I have not gotten any respond to this.. [...] "

cromwellian · 12 years ago

Looks like Apple is using Google Web Toolkit, those are GWT RPC responses. He probably found an CSRF attack.

jchimney · 12 years ago

I read the comments dismissing apples handling of this. What would you have expected them to do? There is a LOT of forensics going on probably even now trying to get a handle on this. A massive corp isn't going to make an announcement until they have some idea what they're talking about. In my books 4 days is a very quick first announcement from a company of this size.

tsm · 12 years ago

These details are befuddling. "Personal information was encrypted and cannot be accessed". It can't be accessed because it's somehow stored elsewhere, or it can't be accessed because of the encryption? That is, does the intruder currently own my encrypted data?

I'm also disappointed that it took them 72 hours to tell us anything, and that the update doesn't even have a timeline for when the site may be back. "Soon" is meaningless.

ajasmin · 12 years ago

Apple should have kept us informed from day one.

But, in their defense it may take days to ascertain what exactly happened.

Once a system is compromised it's nearly impossible to trust anything about it. Auditing the logs, and reviewing the code, crypto, and the mix of platforms they're using (see https://news.ycombinator.com/item?id=6078854) in order to understand what data could be accessed and fix all vulnerabilities is not an easy task.

The PlayStation store was not down for such a long time without reason.

Watabou · 12 years ago

Yeah I'm confused why companies tells us DAYS after something serious happened as opposed to right away. I can understand waiting a day but 3 whole days?! I just don't understand the delay.

It's our data, we should have the right to know what happened to it.

larrys · 12 years ago

"why companies tells us DAYS after something serious happened"

Companies are people. And all the relevant parties involved in handling this may not be accessible to make a decision as quickly as needs to be done. Or at least quickly enough to satisfy all people.

Do you feel you suffered any harm in particular by the delay of three days?

badclient · 12 years ago

It can take more than a day to know what happened to your data.

tlongren · 12 years ago

As I interpreted it, yes, the intruder does have your encrypted data.

num3ric · 12 years ago

I hope you were using good encryption Apple. This guy cracked 400,000 md5 hashed passwords: http://www.youtube.com/watch?v=0WPny7wk960 He says even with password salts, it could be done.