Never mind telnetd. Tier 1 transit providers doing port filtering is EXTREMELY alarming. They have partitioned the Internet, and in a way that automatic routing (BGP) can't get around.
> Tier 1 transit providers doing port filtering is EXTREMELY alarming.
I was admining a small ISP when blaster and its variants hit. Port filtering 139 and the rest was the easiest way to deal with it, and almost over night most of the ISPs blocked it, and we were better for it. There was a time when if you'd put a fresh XP install on the Internet you'd get 5-10 minutes until it would get restarted.
I guess if you're really an admin that needs telnet, you can move it to another port and go around it? Surely you'd tunnel that "old box that needs to stay alive" if that's the usecase? Is there anyone seriously running default telnet on 23 and is really affected by this filtering?
Lots of text games - MUDs - still play over telnet using dedicated MUD clients that implement their own telnet stack. Outright blocking the port has an outsized side efffe on them, this is simply not right.
I run a PDP-10 during the colder parts of the year. It's for historical preservation reasons. There are others doing the same thing. We still offer telnet access because that's how it worked back then. I guess we aren't going to be doing that anymore.
The GP's concern isn't a practical one, it's ultimately about net neutrality. It's not the ISP's job to discriminate against traffic—it's their job to deliver it.
This may seem like a good idea, and frankly is likely a net-positive thing, but it is literally the definition of "ISP decides what apps its customers can and cannot use."
I share the concern and don't really like it either.
Changes like these lend even more credibility to the approach of putting everything on port 443 over TLS, and distinguishing protocols based on hostname / HTTP path.
I do not know what is more critical: the risk of censorship or stand by while hospitals, banking, nuclear power plants and other systems become compromised and go down with people dying because of it. These decision makers not only have powers but also have a responsibility
Have you ever seen a hospital, a bank, a power plan to expose telnetd to the public internet in the last 20 years? It should be extremely rare and should be addressed by company’s IT not by ISPs.
This feels more akin to discovering an alarming weakness in the concrete used to build those hospitals, banks and nuclear power plants – and society responding by grounding all flights to make sure people can't get to, and thus overstress, the floors of those hospitals, banks and nuclear power plants.
Port 23 has been filtered by most providers for decades.
This is why everything converges on using TLS over 443 or a high port number. I don't see this as a huge deal, and especially not one deserving all caps rants about censorship. Save those for things like FOSTA/SESTA.
not to mention, filtering on udp vs tcp, which makes using anything else impossible. Not that I have one, but it's just a bit in a field, why filter on it?
What an amazing bug. I probably spent my first 10 years on the internet just using telnet. They were wild times. You could log ethernet traffic and see passwords. Towards the end of those we started to have a few more single-user machines, but the vast majority were old school many many user machines, where "root" was thought to be tightly restricted (of course, even then, in practice it wasn't if you were in the know).
I never sent root over telnet, but I spent too much vacation time browsing the web via lynx on my school AIX account from a library near my parents' home, because it had a telnet client in addition to the card catalogue program on the otherwise locked down desktop. It was just a more innocent time: you didn't assume your traffic was being logged six ways to Sunday. With telnet access to my AIX account, I could do all the internet things, like mail (pine) and the web (lynx) and irc, from a convenient command line anywhere in the world.
When did we all stop using telnet? I can't even remember. Most of my first 10-15 years was using telnet. One day I used telnet to connect to a shell for the last time and didn't know it. I had a ton of servers all with root telnet access Internet facing. Never hacked once, somehow. Those were the days.
In the Linux / BSD world, SSH took off incredibly fast for the time. I'd estimate that maybe 80% of people had moved to it within the first year of its release.
But adoption stalled when the original SSH moved to a commercial license in 1996-ish - many of us stuck with the last free version, but vulnerabilities started to pile up. There were various half-working alternatives, but it wasn't until OpenSSH came out in 1999 that the remaining telnet holdouts started to move across.
I worked for an ISP in the mid-90s and had been on the Internet since 1989 or so. I recall the progression for me was something like this:
We used telnet in college no problem. It was a fairly well-accepted method of remote access. The heterogeneous network had many different modes, but a major dialup point was the Annex box, which supported telnet into the Unix or VMS machines.
Between Unix machines, we would often prefer "rlogin" instead. There were several horrific iterations of other remote-access protocols such as "remsh". rlogin was notorious for its "/etc/hosts.equiv" authorization method which trusted DNS and should've been perceived as Swiss Cheese from the outset. rlogin was, IIRC, directly related to rsh and rcp and used the same frameworks. rlogin was no more secure than telnet, but probably less secure because of its conveniences.
We also used port 23/tcp for remote management, for example Cisco routers. They weren't running telnetd, but it was the port where you connected remotely and logged in with or without credentials.
rlogin persisted alongside telnet, until encryption came into fashion and ssh was distributed. Once ssh was available and working well, everyone knew that telnetd and rlogind were on borrowed time. The services were shut down and disabled in inetd. The ports were sometimes blocked. Security advisories went out.
I suppose it took a long, long time for ssh to finally dominate, and for people to abandon telnetd mostly, but it was fairly thorough. We all recognized the superiority of sshd's authentication and encrypted channels.
There were mitigations for people to extend their legacy use of telnetd and rlogind. For example, tcp wrappers and fail2ban could be implemented. Firewall filters could select only authorized networks. VPNs could tunnel through an Intranet that still used them. So, the services lived on wherever they didn't need to be exposed on the public Internet. But I think most Unix admins got the picture by the end of the dot-com bubble.
It's hilarious, especially given that I have memories of similar rlogin vulnerabilities -- various unixes being vulnerable to rlogin -l '-froot' in the 90s.
Never used telnet to log in to something but it is a cool debugging tool, so used it for that. E.g. can this container even send traffic to that container at all.
So Telnet as a client is not dead though, right? A long time ago, I used to use the Telnet client to talk to SMTP servers (on port 25) and send spoofed emails to friends for fun.
With port blocking widening in scope, I’ve long believed that we would one day have every service and protocol listening on port 443. Since all other ports are being knocked off in the name of security, we’ll end up having one port that makes port based filtering useless.
I've never really understood why it's a thing to use a telnet client for transmitting text on a socket for purposes other than telnet. My understanding is that telnet is a proper protocol with escape sequences/etc, and even that HTTP/SMTP/etc require things like \r\n for line breaks. Are these protocols just... close enough that it's not a problem in practice for text data?
If it's alright to be pedantic, anyone with programming knowledge can do the same without these tools. What these offer is tried and tested secure code for client side needs, clear options and you don't need to hand roll code for.
I don’t remember how I did it but when I was about 12 years old I somehow managed to send SMS from Telnet to cell phones, and to the receiver they appeared to be sent by an official Telecom account - good that I was still an innocent child, had I discovered this a few years later I may have tried doing something nefarious with it.
None of this affects the use of telnet the client program nor the ability to run a telnetd on your own host (but do be sure it's patched!).
What's happened is that global routing on the internet (or big chunks of it, it's not really clear) has started blocking telnet's default port to protect presumably-unpatched/unpatchable dinosaur systems from automated attack. So you can no longer (probably) rely on getting to a SMTP server to deliver that spoofed email unless you can do it from its own local environment.
You would still be able to use the telnet client to connect to an SMTP server on TCP port 25, just not port 23, right? I don't think that part changed here.
(Remember hearing about this a long time ago (from some searching I think it was in 1999 via Slashdot) and verified some instance of it still exists/works.)
Telnet is used in legacy, IoT, embedded, and low-level industrial hardware. It's also intentionally enabled on devices where automation was written for telnet and it wasn't easy to switch to ssh.
If you investigate most commercial uses of ssh, the security is disabled or ignored. Nobody verifies host keys, and with automation where hosts cycle, you basically have to disable verification as there's no easy way around the host keys constantly changing. Without host key verification, there's kinda no point to the rest.
Even assuming the host keys were verified, the popular ssh conventions are to use either long-lived static keys (and almost nobody puts a password on theirs), or a password. Very few people use SSH with 2FA, and almost no-one uses ephemeral keys (OIDC) or certificates (which many people screw up).
So in terms of how people actually use it, SSH is one of the least secure transport methods. You'd be much more secure by using telnet over an HTTPS websocket with OAuth for login.
How do you automate, for example, "HTTPS over websocket with OAuth", without providing some kind of hard-coded, static or otherwise persistent authentication credentials to the calling system in some form (either certificate based auth, OAuth credentials, etc.)?
The problem with IoT and embedded secrets isn't really a solved problem, from what I can tell. I'm not sure that OAuth exactly solves the problem here. Though all your comments about SSH (especially host verification) holds true.
Just honestly trying to understand the possible solution space to the IoT problem and automated (non-human) authorization.
Unless you manage to leak your private host/client SSH keys, this is close to being as secure as it gets.
I'd say that HTTPS (or TLS in general) is more problematic, since you need to trust numerous root CAs in machine/browser store. Sure, you can use certificate pinning, but that has the same issues as SSH host key verification.
The known_hosts file is verification of host keys. It's not verification of a host cert, which is a different thing. Most sshd instances are running on ad hoc hardware without the ability to associate them with someone a cert authority would be willing to authenticate.
Basically people running services that need cert-based authentication are already using TLS (or if they're using sshd they've locked it down appropriately). SSH is for your workstation and your RPi and whatnot.
> IMHO we need a good telnet replacement that sends signed data. Most people interpret signatures as allowed under FCC rules, just not encryption.
I know from bitter experience that IPsec is a “now you have two problems” kind of solution, but the Authentication Header is a thing and is supported by most (all?) implementations. Ham radio operators probably don’t have much use for the actual features of telnet compared to plain netcat, do they? (It’s mostly terminal feature negotiation and such.)
I've always used ssh to connect to it. And it's true that their port 23 is still open at last check. If you cannot reach port 23, and you irrationally hate ssh, you may use 14321 as an alternate.
MUDs were my introduction to telnet- I grew up a university kid and had access to Wesleyan's minicomputer EAGLE.WESLEYAN.EDU running OpenVMS. I used it to telnet to CMU's TinyMUD and later other TinyMUDs around the country. I recall OpenVMS's telnet had a problem with newlines/carriage returns so all the text was staircased, so I ended up learning C and writing a MUD client. I still habitually use telnet today even if netcat and many other tools have replaced it.
All of that was foundational for my career and I still look back fondly on the technology of the time, which tended to be fairly "open" to exploration by curious-minded teenagers.
Probably one of the reasons this bug survived so long is that it isn't used much for priveleged access any more, but so you can play a moo or play you an ASCII movie, as people below you are replying.
On the bright side that CVE seems like pretty great news for the hardware hacking community hoping to get root on embedded devices which have open telnetd.
> Someone upstream of a significant chunk of the internet’s transit infrastructure apparently decided telnet traffic isn’t worth carrying anymore. That’s probably the right call.
Does this impact traffic for MUDs at all? I know several MUDs operate on nonstandard Telnet ports, but many still allow connection on port 23. Does this block end-to-end Telnet traffic, or does it only block attempts to access Telnet services on the backbone relays themselves?
MUDs use plaintext TCP protocols that are accessible to a wide range of clients.
The Telnet protocol is well-defined and not completely plaintext. There are in-band signaling methods and negotiations. Telnet is defined to live on 23/tcp as an IANA well-known, privileged, reserved port.
MUDs do none of this. You can usually connect to a MUD using a Telnet client, but most players hate the experience and often deride this method in favor of a dedicated, programmable client.
The fact that MUDs inhabit higher 4-digit ports is an artifact from their beginnings as unprivileged, user-run servers without a standardized protocol or an assigned “well-known port” presence. If you want your MUD to be particularly inaccessible, you could certainly run on port 23 now!
As a MUD enthusiast of two decades, this is not accurate. Where are you getting this information?
Most MUDs implement RFC 854, and a number of non-standard Telnet option subnegotiation protocols have been adopted for compression (MCCP2), transmission of unrendered data (ATCP, GMCP, ZMP), and even a mechanism for enabling marking up the normal content using XML-style tags (MXP). These telopts build on the subnegotiation facility in standard Telnet, whose designers knew that the base protocol would be insufficient for many needs; there are a great number of IANA-controlled and standardized telopt codes that demonstrate this, and the MUD community has developed extensions using that same mechanism.
> You can usually connect to a MUD using a Telnet client, but most players hate the experience and often deride this method in favor of a dedicated, programmable client.
I think you are confusing "telnet" the program with "telnet" the protocol. I am speaking here of the protocol, defined at base in RFC 854, for which "telnet" the program is but one particularly common implementation. You look at any of those "dedicated, programmable clients" and they will contain an implementation of RFC 854, probably also an implementation of RFC 1143 (which nails down the rules of subnegotiation in order to prevent negotiation loops), and an implementation of the RFCs for several standard telopts as well as non-standardized MUD community telopts. I can speak for the behavior of MUSHclient in especial regard here, though I am also familiar with the underlying Telnet nature of Mudlet, ZMud, and CMUD, not to mention my very own custom-made prototype client for which I very much needed to implement Telnet as described above.
Wouldn't that imply that >80% of all monitored telnet sessions were exploit attempts for the specific CVE in question? Even with the scale of modern botnets, that seems unrealistic for a single vuln that was undisclosed at the time.
Wouldn't attention to getenv() calls yield more benefit? Such calls are where input typically isn't parsed--because parsing is "hard"--becoming targets for exploit.
The present fix is to sanitize user input. Does it cover all cases?
Readit News
I was admining a small ISP when blaster and its variants hit. Port filtering 139 and the rest was the easiest way to deal with it, and almost over night most of the ISPs blocked it, and we were better for it. There was a time when if you'd put a fresh XP install on the Internet you'd get 5-10 minutes until it would get restarted.
I guess if you're really an admin that needs telnet, you can move it to another port and go around it? Surely you'd tunnel that "old box that needs to stay alive" if that's the usecase? Is there anyone seriously running default telnet on 23 and is really affected by this filtering?
This may seem like a good idea, and frankly is likely a net-positive thing, but it is literally the definition of "ISP decides what apps its customers can and cannot use."
I share the concern and don't really like it either.
This is still true, though 5-10 minutes is slightly pessimistic. Source: https://youtu.be/6uSVVCmOH5w
TL;DW - Guy installs XP and makes it internet accessible, only takes 15 minutes before the first malware appears on it.
Deleted Comment
Filtering one port is not censorship. Not even close.
Great analysis, thank you!
New thread: Reports of Telnet's Death Have Been Greatly Exaggeratedhttps://news.ycombinator.com/item?id=46980355
This is why everything converges on using TLS over 443 or a high port number. I don't see this as a huge deal, and especially not one deserving all caps rants about censorship. Save those for things like FOSTA/SESTA.
Is it on port 23/tcp, and what are the ASNs?
The report specifically says that cloud networks like VPS, AWS seemed exempt.
Anyway, just wild seeing this:
> telnet -l 'root -f' server.test
or
> USER='-f root' telnet -a server.test
Survive 11 years.
But adoption stalled when the original SSH moved to a commercial license in 1996-ish - many of us stuck with the last free version, but vulnerabilities started to pile up. There were various half-working alternatives, but it wasn't until OpenSSH came out in 1999 that the remaining telnet holdouts started to move across.
We used telnet in college no problem. It was a fairly well-accepted method of remote access. The heterogeneous network had many different modes, but a major dialup point was the Annex box, which supported telnet into the Unix or VMS machines.
Between Unix machines, we would often prefer "rlogin" instead. There were several horrific iterations of other remote-access protocols such as "remsh". rlogin was notorious for its "/etc/hosts.equiv" authorization method which trusted DNS and should've been perceived as Swiss Cheese from the outset. rlogin was, IIRC, directly related to rsh and rcp and used the same frameworks. rlogin was no more secure than telnet, but probably less secure because of its conveniences.
We also used port 23/tcp for remote management, for example Cisco routers. They weren't running telnetd, but it was the port where you connected remotely and logged in with or without credentials.
rlogin persisted alongside telnet, until encryption came into fashion and ssh was distributed. Once ssh was available and working well, everyone knew that telnetd and rlogind were on borrowed time. The services were shut down and disabled in inetd. The ports were sometimes blocked. Security advisories went out.
I suppose it took a long, long time for ssh to finally dominate, and for people to abandon telnetd mostly, but it was fairly thorough. We all recognized the superiority of sshd's authentication and encrypted channels.
There were mitigations for people to extend their legacy use of telnetd and rlogind. For example, tcp wrappers and fail2ban could be implemented. Firewall filters could select only authorized networks. VPNs could tunnel through an Intranet that still used them. So, the services lived on wherever they didn't need to be exposed on the public Internet. But I think most Unix admins got the picture by the end of the dot-com bubble.
With port blocking widening in scope, I’ve long believed that we would one day have every service and protocol listening on port 443. Since all other ports are being knocked off in the name of security, we’ll end up having one port that makes port based filtering useless.
As are many other tools. But the ones above are basically far better direct telnet alternatives.
What's happened is that global routing on the internet (or big chunks of it, it's not really clear) has started blocking telnet's default port to protect presumably-unpatched/unpatchable dinosaur systems from automated attack. So you can no longer (probably) rely on getting to a SMTP server to deliver that spoofed email unless you can do it from its own local environment.
But that's 23 and smtp is 25.
(OK, I know one ancient talker that uses it - but on a very non-standard port so a port 23 block wouldn't be relevant)
telnet towel.blinkenlights.nl https://www.youtube.com/watch?v=Mhcf6tc2jeQ
(Remember hearing about this a long time ago (from some searching I think it was in 1999 via Slashdot) and verified some instance of it still exists/works.)
If you investigate most commercial uses of ssh, the security is disabled or ignored. Nobody verifies host keys, and with automation where hosts cycle, you basically have to disable verification as there's no easy way around the host keys constantly changing. Without host key verification, there's kinda no point to the rest.
Even assuming the host keys were verified, the popular ssh conventions are to use either long-lived static keys (and almost nobody puts a password on theirs), or a password. Very few people use SSH with 2FA, and almost no-one uses ephemeral keys (OIDC) or certificates (which many people screw up).
So in terms of how people actually use it, SSH is one of the least secure transport methods. You'd be much more secure by using telnet over an HTTPS websocket with OAuth for login.
The problem with IoT and embedded secrets isn't really a solved problem, from what I can tell. I'm not sure that OAuth exactly solves the problem here. Though all your comments about SSH (especially host verification) holds true.
Just honestly trying to understand the possible solution space to the IoT problem and automated (non-human) authorization.
I'd say that HTTPS (or TLS in general) is more problematic, since you need to trust numerous root CAs in machine/browser store. Sure, you can use certificate pinning, but that has the same issues as SSH host key verification.
PCI DSS, HIPAA, and ISO 27001 each either highly recommend or enforce this.
I wouldn't use a jumphost without it.
The known_hosts file is verification of host keys. It's not verification of a host cert, which is a different thing. Most sshd instances are running on ad hoc hardware without the ability to associate them with someone a cert authority would be willing to authenticate.
Basically people running services that need cert-based authentication are already using TLS (or if they're using sshd they've locked it down appropriately). SSH is for your workstation and your RPi and whatnot.
IMHO we need a good telnet replacement that sends signed data. Most people interpret signatures as allowed under FCC rules, just not encryption.
I know from bitter experience that IPsec is a “now you have two problems” kind of solution, but the Authentication Header is a thing and is supported by most (all?) implementations. Ham radio operators probably don’t have much use for the actual features of telnet compared to plain netcat, do they? (It’s mostly terminal feature negotiation and such.)
I'm breaking a tonne of FCC rules right now.
https://www.alt.org/nethack/
All of that was foundational for my career and I still look back fondly on the technology of the time, which tended to be fairly "open" to exploration by curious-minded teenagers.
The problem is the auth is plain text too and you're open to having your credentials stolen.
I really should update it to allow more secure options
Not anymore ;)
Seriously though: did you notice any spikes up or down?
If you'd run it on a non-standard port, anyone can still connect with netcat, socat, etc etc.
Deleted Comment
Deleted Comment
It seems to use a different telnetd (busybox?), because from what I can tell it's not prone to this error.
Does this impact traffic for MUDs at all? I know several MUDs operate on nonstandard Telnet ports, but many still allow connection on port 23. Does this block end-to-end Telnet traffic, or does it only block attempts to access Telnet services on the backbone relays themselves?
MUDs use plaintext TCP protocols that are accessible to a wide range of clients.
The Telnet protocol is well-defined and not completely plaintext. There are in-band signaling methods and negotiations. Telnet is defined to live on 23/tcp as an IANA well-known, privileged, reserved port.
MUDs do none of this. You can usually connect to a MUD using a Telnet client, but most players hate the experience and often deride this method in favor of a dedicated, programmable client.
The fact that MUDs inhabit higher 4-digit ports is an artifact from their beginnings as unprivileged, user-run servers without a standardized protocol or an assigned “well-known port” presence. If you want your MUD to be particularly inaccessible, you could certainly run on port 23 now!
Most MUDs implement RFC 854, and a number of non-standard Telnet option subnegotiation protocols have been adopted for compression (MCCP2), transmission of unrendered data (ATCP, GMCP, ZMP), and even a mechanism for enabling marking up the normal content using XML-style tags (MXP). These telopts build on the subnegotiation facility in standard Telnet, whose designers knew that the base protocol would be insufficient for many needs; there are a great number of IANA-controlled and standardized telopt codes that demonstrate this, and the MUD community has developed extensions using that same mechanism.
> You can usually connect to a MUD using a Telnet client, but most players hate the experience and often deride this method in favor of a dedicated, programmable client.
I think you are confusing "telnet" the program with "telnet" the protocol. I am speaking here of the protocol, defined at base in RFC 854, for which "telnet" the program is but one particularly common implementation. You look at any of those "dedicated, programmable clients" and they will contain an implementation of RFC 854, probably also an implementation of RFC 1143 (which nails down the rules of subnegotiation in order to prevent negotiation loops), and an implementation of the RFCs for several standard telopts as well as non-standardized MUD community telopts. I can speak for the behavior of MUSHclient in especial regard here, though I am also familiar with the underlying Telnet nature of Mudlet, ZMud, and CMUD, not to mention my very own custom-made prototype client for which I very much needed to implement Telnet as described above.
MOOs do.
Since Telnet is totally plain text that would absolutely be easy to do right?
That said in this day and age, servers on the public network really ought to use SSH.
It's crazy to think that some dude is singlehandedly responsible for ultimately ending the telnet era in such a definitive way.
One for the history books.
Well, one person to put up the PR and one dude to approve it - back in 2015. It isn't the security researcher's fault.
https://codeberg.org/inetutils/inetutils/commit/fa3245ac8c28...
One of the changes is:
What is the reason for a rename these days? If I saw that in a code review I’d immediately get annoyed (and probably pay more attention)Deleted Comment
The present fix is to sanitize user input. Does it cover all cases?