The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse.
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
I have to admit - I still grind my teeth every time I see "dns propagation" used without a direct follow-up that it's a myth, you're looking at cascading cache expiry.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
DNS changes propagate. They just do-so in a pull, not push, way.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
Yes! The idea of DNS records "propagating" gave me entirely the wrong mental model of DNS very early in my career. Granted, the confusion didn't last long because I read the cricket book soon after, but it was still pretty jarring.
> Without DNS, you'd need to memorize IP addresses for every website.
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
It might be worth mentioning the concept of "stub resolver" and clarifying a bit that a nameserver is a resolver. That might be being pedantic, but thought it might be worth clarifying that the difference conceptually may just be what the particular dns server answering the query is authoritative for, if anything.
One other thing that might be worth a mention is the concept of the OS' resolver and "suffix search order", with an example of connecting (https, ping, ssh, whatever protocol) to a host using just the hostname, and the aforementioned mechanism that (probably) allows this to connect to the FQDN you want. (Also, now that I type that, do you mention "FQDN" at all? If not, maybe should.)
On that note one final thought that occurs to me is the error/confound that may occur if a hostname is entered and is not resolved, but does resolve with one of the domain suffixes attached on a retry (particularly can be confusing with a typo coupled with a wildcard A record in a domain, for example.) I recognize that the lines that look like DNS records are not explicitly stated to be in a format for any particular dns server software, and even if they were, they're snippets without larger context so we don't know what the $ORIGIN for the zone might be, an adjacent concept you might want to explore, even if just for your own edification is that of the effect of a terminating "." at the end of a hostname, either at resolution or configuration time.
Just offering feedback that might help you add to the article.
> You can change which resolver you use in your network settings. I switched to 1.1.1.1 on my machines - it's noticeably faster than my ISP's default resolver.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
I think CNAMES were a mistake, think of them as lightweight NS records. It is a simple protocol solving a rather complex system. I have respect for simple things I have tried implementing my own server, did not go well.
You have to go back five decades if you want to use a host file. My point is that cache and making are the three hard things in computer science DNS has had both problems several times. I choose DNS over hosts files every time.
Only oddity was the reference to the "router cache". I agree if your browser tried to lookup example.com the local cache would be used, but then it would be the system's configured DNS server - and that would most likely be an ISP, rather than your local router.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
I think this is probably quite dependent on what’s normal for ISPs in the region. In the UK for example, every ISP router I’ve had runs a DNS server and it’s that which is given out via DHCP. It then forwards onto the ISPs DNS platform.
I would argue the contrary - most home routers are running a DNS server of some kind. They forward to upstream, but will resolve local names like your printer and whatnot.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.
IIRC a resolver is what people would think of as a DNS server only it's not an authority for any domains. Like you said, they're used to get load off of authoritative servers and are very common. I think dnsmasq is mentioned explicitly in the O'Reilly locust book but it's been a while.
My parents are with Bell (the biggest ISP in Canada) and use the Bell Gigahub (Router/AP/Switch in one). It does have a DNS cache and the its set as the DNS resolver in their DHCP configuration.
Readit News
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
There is a reason we say "it's always DNS".
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
you mean DNSSEC, right? RIGHT?
DNSSEC at least has its own RFC and uses the TXT record, which was added to DNS specifically for this kind of use case.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
And checkout their Mess with DNS playgound!
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
The fact that a server can serve multiple vhosts and do TLS cert selection via SNI is not related to the lookup of what server to connect to.
One other thing that might be worth a mention is the concept of the OS' resolver and "suffix search order", with an example of connecting (https, ping, ssh, whatever protocol) to a host using just the hostname, and the aforementioned mechanism that (probably) allows this to connect to the FQDN you want. (Also, now that I type that, do you mention "FQDN" at all? If not, maybe should.)
On that note one final thought that occurs to me is the error/confound that may occur if a hostname is entered and is not resolved, but does resolve with one of the domain suffixes attached on a retry (particularly can be confusing with a typo coupled with a wildcard A record in a domain, for example.) I recognize that the lines that look like DNS records are not explicitly stated to be in a format for any particular dns server software, and even if they were, they're snippets without larger context so we don't know what the $ORIGIN for the zone might be, an adjacent concept you might want to explore, even if just for your own edification is that of the effect of a terminating "." at the end of a hostname, either at resolution or configuration time.
Just offering feedback that might help you add to the article.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
https://github.com/cleanbrowsing/dnsperftest
I have been broken for three decades and I still don't understand DNS. It is a simple protocol but people use it in complicated manners.
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
IP's can change without warning.
EDIT I might be off by one here.
But it's not an issue at all, and it provides a convenience that can be depended on by a lot of your dependencies.
Code may use domains instead of ip addresses (which provides resiliency), package managers like apt depend on domains. And so on.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
Nowadays I'm in Finland and definitely the router runs no DNS service, the DHCP service advertises the ISP resolvers.
Probably depends on the region/ISP I guess, but I had no expectation that it would be the more common option.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.