I wonder if they are using tooling that doesn't properly retain DNSKEY records for DS that recently removed? This is one of the reasons we perform controlled automated key rotation and removal in DNSimple, so that we can ensure we retain the keys in the authoritative zone on each key rollover giving the DS records time to expire from caches.

It was especially bad since their status page wouldn't even resolve! I eventually just restarted my local caching DNS server.

We had a DNS related outage with route53. Some of our zones just lost some records and then they reappeared. Could that explain what happened to slack's DNSSEC related records?

Aren't you, in fact, the same Thomas Ptacek who has repeatedly claimed that DNSSEC is so irrelevant that events like this would go essentially unnoticed?

Edited to add, e.g. https://news.ycombinator.com/item?id=22400167

> DNSSEC is moribund and almost nobody uses it; in reality, the DNSSEC root private keys could land on Pastebin tomorrow and nothing would "break"

We have this whole thread here about a "service disruption" for Slack, and nobody leaked the "root private keys" just one person made a dumb error and it blew up their site.

Unlikely to help in this particular case (which is a root-level DS record).

Awesome handy tip, thanks!

Great analysis, thanks!

Slack support says that users should tell their ISPs to invalidate the DNS cache for slack.com https://status.slack.com/2021-09/06c1e17de93e7dc2 (access with 8.8.8.8 as resolver - fallback https://slack-status.azureedge.net/)

Since the faulty DS record was in .com, everyone has a max wait-for-ttl-to-expire time of 24h.

Google/Cloudflare etc. seem to also invalidate .com caching very quickly, 8.8.8.8 quickly was the first workaround.

Meanwhile, 14 hours later, DTAG in Germany still does not resolve. The default resolvers have dnssec enabled.

dig slack.com +cd

tells the resolver to skip dnssec validation tests, and then it works again. Screenshots with the command output in https://twitter.com/dnsmichi/status/1443840645513293853?s=2

Very interested in the post-mortem analysis. I think there were similar mistakes as with nasa.gov incident and the comcast analysis in 2012: https://www.internetsociety.org/blog/2012/01/comcast-release...

Learnings for me:

- dnstracer (https://gitlab.com/dnsmichi/dotfiles/-/blob/main/Brewfile#L5...) helps with detecting missing glue records, but not dnssec

- dnstrace (https://github.com/rs/dnstrace) is a better alternative with dnssec

Nope, Level 3's DNS servers don't do DNSSEC validation, as can be tested with dig:

    % dig www.brokendnssec.net +dnssec @4.2.2.1
    
    ; <<>> DiG 9.10.6 <<>> www.brokendnssec.net +dnssec @4.2.2.1
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58098
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags: do; udp: 8192
    ;; QUESTION SECTION:
    ;www.brokendnssec.net.  IN A
    
    ;; ANSWER SECTION:
    www.brokendnssec.net. 300 IN A 104.22.35.212
    www.brokendnssec.net. 300 IN A 172.67.36.129
    www.brokendnssec.net. 300 IN A 104.22.34.212
    
    ;; Query time: 84 msec
    ;; SERVER: 4.2.2.1#53(4.2.2.1)
    ;; WHEN: Thu Sep 30 17:59:35 MDT 2021
    ;; MSG SIZE  rcvd: 97
    
    % dig www.brokendnssec.net +dnssec @8.8.8.8
    
    ; <<>> DiG 9.10.6 <<>> www.brokendnssec.net +dnssec @8.8.8.8
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 22514
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags: do; udp: 512
    ;; QUESTION SECTION:
    ;www.brokendnssec.net.  IN A
    
    ;; Query time: 96 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Thu Sep 30 17:59:44 MDT 2021
    ;; MSG SIZE  rcvd: 49

Interdependencies among services can cause all sorts of unexpected issues.

It seems to be standard practice to be as fuzzy as possible during an outage, and only share details later. Probably avoids anyone looking stupid if the initial hunch turns out to be wrong.

Good tip, back up, appreciate it