I have been for like a month now noodling on a long-form post about a piece of software we've been noodling with for 3-4 years now. Kurt is freaking out, because we haven't written anything on the blog since, like August. Finally I'm like, look, I will write the simplest thing I can come up with. We'll do the opposite of what we've been doing. We'll do anti- effortposts. I bet I can do one in 30 minutes.
I promise, I thought about this less than you have. It's a thing we were tinkering with, and I wrote about it. That's all.
Thanks for the post. I see now that it's bananas, but I didn't quite get that from the blog post. You list the things this agent can do, but I immediately assumed it went in the other direction, because anything else would be bananas. The statement in the README put it more succinctly for me:
> A compromised remote could use the VS Code Remote connection to execute code on your local machine.
I feel like that "security note" should have a CVE number next to it.
That first paragraph (in your HN comment, not the blog post) would have read better with zero periods.
Glad to see one of my favorite blogs is still active. I was starting to get a little worried.
The first two blogs currently listed, McCord-Valim’s FLAME-Livebook-GPU and your post which has the word “murid”, truly captures the psychological arc of a developer.
It seems like ssh may be the problem. I feel like there should be a way to ask for a docker like experience when ssh ing. Tell it to use is specific APIs to prevent process or filesystem access outside a particular folder (possibly allowing system binaries, although that might complicate things and require vscode to push more stuff to client). Basic googling indicates some choot options in the ssh server but nothing is mentioned on 5he ssh client man page.
Although maybe the solution is to download a docker container on the remote (directly if it has access to dockehub or over an ssh connection if it doesn't) then run a docker container mounting the remote directory and connect to it over ssh.
The problem with only syncing files from the subdirectory is that you also want to be able to run &debug on the remote started by vscode. So plugins also need access to remote or to run remotely for some code observation that may be prohibitively expensive if run locally(pre sync of whole subdir)
If you’re ssh-ing as a normal user you should only be able to modify files in that users remote home directory. You don’t need all that docker stuff just stop using admin accounts for normal access.
The more I learn about how VSCode works the more it seems like it's held together with duct tape and the most cursed ideas a JS developer could come up with.
Even just from the SSH extension - the workspace URIs have two formats: essentially just the hostname and hex-encoded JSON documents. The latter case happens when additional info is needed, e.g. specific username, or... The hostname includes an uppercase letter.
Which is actually necessary because when they're saved to recent workspaces they're lowercased for whatever reason.
The SSH connections also support configuring extensions that are to be installed on the server, but don't go too crazy with it or you won't be able to connect to Windows hosts, since they're passing them ass command line arguments via CMD, which has a 8191 character limit (they're using CMD to call... PowerShell...).
I don't know, I found VS Code to be nicer than Eclipse. I've never had a need for SSH through an IDE, so I can't speak to that. Usually I'd just SSH through putty and use Vi if I need to do stuff on the server.
I like it because it’s really easy to extend the editor with custom language support/tooling if you know JavaScript/Typescript. Providing custom completions, diagnostics, etcetera. I can also provide custom Go to definitions for cross-language support.
This is going to sound naive but, I don't understand what the security issue is. If you can ssh into a machine and port forward a socket, you already have permission to do all the other things. VSCode's protocol seems to be exposing it in way that's more convenient for them.
How is this a security problem? Is it because someone on the same network as the remote machine but without SSH access can connect the port that is forwarded over SSH?
As a user, I quite like how well VSCode's SSH system works.
I think the difference is that what VSCode is doing is not an SSH Session like you get in a terminal with the ssh command or putty.
VSCode is installing a remote agent on the target machine that happens to use ssh as its transport protocol, and offers to share that transport with the user.
Is this a problem? Not if it only does things you want it to do. However any agent based system exposing an arbitrary API is suddenly a much bigger attack and risk surface area than the well trod (and still fraught) path of emulating a terminal over ssh.
> However any agent based system exposing an arbitrary API is suddenly a much bigger attack and risk surface area than the well trod (and still fraught) path of emulating a terminal over ssh.
I can see how this increases local (to the remote system) attack surface, but as long as the agent has the same OS privileges as the user logged in over SSH, what extra remote risk does this introduce?
I think the latest Jetbrains tools do the same or similar things. They also install their own server on the target machine and connect to that. And I mean it's Jetbrains, so again, closed source tools. But it's not Microsoft so nobody is talking about it I guess.
Is it better to programmatically interact with bash to provide the features VSCode does? Do note that I am unwilling to accept an implementation with less features/ease of use!
I can see how writing a custom agent that provides remote access to privileged API's is a bad idea but bash isn't exactly the most secure piece of software in the world.
This is really big. I always thought Vscode ssh capability was just some black magic that the open source community wasn't able to replicat well with vscodium extensions. Yet, it seems the reason why is that MS threw a curve ball with that naming.
> The agent runs over port-forwarded SSH. It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can:
- Wander around the filesystem
- Edit arbitrary files
- Launch its own shell PTY processes
- Persist itself
When you ssh into a remote server as a client, afaik that server cannot execute arbitrary code on the client. At a minimum, the client would have to explicitly take action for that to happen.
I think you’re basically right: it’s not a security problem in that it doesn’t inherently traverse any vulnerabilities or security boundaries.
But it is a security problem in the same way that “curl | bash” is a security problem. An even closer equivalence might be something like “curl | bash” in your bashrc.
You're already trusting that third party agent on your own computer. If VSCode itself was malware then it can do anything you can do, including sshing into remote machines and running commands behind your back.
It's helpful for evading detection, because if you've compromised a machine, you can drop in the server binary and it'll have been added to the allowlist for devs to run.
> If you can ssh into a machine and port forward a socket, you already have permission to do all the other things.
Only technically. There are lots of situations where people (or robotic alarm bells) will be miffed if you instal your custom sw stacks, remote shells, "i don't know what these binaries are" etc on some servers you're supposedly just editing config files on.
There are lots of situations where people will not notice at all. At my last gig, I wrote pretty extensive Ansible manifests that built and configured all my favorite tools on on bog-standard company provided development hosts so I found them cozy for remote work. I took severe liberties with these hosts and nobody batted an eye.
A piece of software that is ostensibly trying to help you is gathering information without your knowledge and.. one of these days may upload it to the mothership? That's kinda not ok don't you think?
One reason it's a problem is because it's very unexpected behavior. Software should follow some sort of pattern that you can develop a reasonable mental model around.
But the author doesn’t give a single example, and so there’s nothing someone else can do to explain why or continue the conversation. The best we got use that he has a vague sense of unease, which I don’t think is very useful.
...which I've come to expect from Microsoft. Windows scrozzles the efi partition for my other OS every third or fourth time it installs updates. It's like they think they own every computer they have code on.
I ran the servers for our networks, binary exploitation, and intro systems programming classes, and this thing is a major annoyance. It is because of this stupid RAT that students do not understand how to use the OpenSSH client.
I've tried a few things to fix this:
1., I set a motd on our class servers asking the students not to use the VSCode remote server plugin.
2. I ran `ncdu /home` in front of the class and demonstrated how, without exception, everyone with >100MB of disk usage on the class server was a VSCode user.
3. I set a user process limit of 45, because the VSCode RAT (somehow) uses ~50 Node processes. When students inevitably ignored the motd and the in-class warning, they hit the process cap and had to ask us to kill their processes in order to get back onto the system.
4. I replaced the process limit with a script that kills all the .vscode-server RATs every 10 seconds.
Your comment is bringing back so many memories of being in university and working around the anachronistically strict limits the uni sysadmins had on the network
Ah yes, one of my favorite skills learned in school was how to work around arbitrary rules made up by cranky sysadmins! I still use it all the time.
The smart kids (whom I’m sure look like they are learning that precious openssh client) are doing their assignment locally or on a free tier VPS with VS Code and scping the thing over when it’s done.
They’re also smart enough to learn openssh when they need it IRL.
This isn't something specific to VSCode being popular. When I was in college over a decade ago, there were students using Sublime with an SFTP plugin, coding locally and copying stuff with FileZilla or other similar GUI clients to transfer files (I distinctly remember seeing some program that was named something duck-related but for the life of me I can't remember any other specifics). Sublime in particular could be frustrating in the class I TA'd where they had some assignments to deal with processing some (very basic) machine code to simulate running the tool the class used to assemble and run the code from the (similarly basic) ISA they had been taught, which involved them using hexdump or something similar to understand how the bytes in the files worked, but Sublime "helpfully" would render the object files as as their text representation of the hexdump, with extra spacing for readability and swapped endianness compared to the way the bytes would show up if hexdumped on the school's Linux server. There would always be several students every semester who would show up at office hours unable to figure out why their code that was written to try to read an ASCII string like "AD DE EF BE " but instead would instead find some text they didn't recognize because they didn't think to look at the actual byte values, which would always just happen to start with 0xDE, 0xAD, 0xBE, and 0xEF.
The issue is not about live editing on servers, it’s about running a process out on the server that spawns,according to GP, 50-some processes. VS Code remote editing and Sublime/any SFTP/SCP-style deploy are on completely different levels.
Can you explain why though? I get that you had to put a lot of effort into blocking VSCode but it's not clear what have VSCode caused to motivate this.
It sounds like they’re operating at larger scales, with a large number of users sharing a resource. Each user spawns 50 processes and consumes 100MB of space if they’re using VSCode. Assuming a low count of 1000 users, that’s 50K running processes and 100GB of space consumed. To enable a text editor.
> 1., I set a motd on our class servers asking the students not to use the VSCode remote server plugin. ...When students inevitably ignored the motd...
The provider of the servers? If the students want to use their own server, I’m sure they’re free to do whatever they want. Also VS code has plug-ins that just SFTP the files up.
I don't know what the alternative would be here. VSCode's SSH edit works surprisingly well, and I stopped fiddling with vim, nano, or micro on my remote machines long ago.
The agent gets out of my way and lets me work in peace. It almost feels like I'm working from my local machine—a huge win in my book.
This could be a security risk, but the development experience is unmatched. I couldn't care less which other editor VSCode is killing, as I mostly don't care about my tools as long as they get out of the way and let me do my job.
The alternative is suggested by tramp, which from what I know treats the remote as a network filesystem instead of an execution host. I don't believe that tramp deploys any binaries: it reads and writes bytes over pipes and all meaningful execution happens locally. Notably, it does not achieve persistence, because there's a difference between "VSCode plugins have access while you're SSH'd in" and "VSCode plugins have access forever".
Persistence is important to me, and making it read-only significantly reduces its usefulness. I regularly SSH into a dev machine to run scripts and update configurations. As long as a tool lets me do that without getting in my way, I'm good with any solution that works.
That's an issue with any plugin system, right? AFAIK no IDE has a plugin system with capabilities or a sandboxed interpreter.
VSCode does have a thing where it's like do you trust the authors of this project. Not sure what it does because I've never had to use it. From StackOverflow[1]:
>If you select No, I don't trust the authors, Visual Studio Code will open the workspace in 'restricted mode'. This is the default for all new workspaces. It lets you safely browse through code but disables some editor feature, including debugging, tasks, and many extensions. However, keep in mind that 'restricted mode' is all you need for many use cases.
Actually if restricted mode[2] is any good, vscode might be better at security than most other editors/IDEs.
Yeah, I'm all in for a more secure option as long as it allows me to do everything that VSCode's SSH agent does. But if the devex goes down the drain because of "security" then I'm good for now.
From what I’ve seen, my colleagues using vscode are hampered in ways they don’t recognize because they have no concept of how good things could be. Working with multiple remotes, they’re often completely unaware of which one they’re connected to or what state their connection is in. Their terminals are laggy and session state persistence is inconsistent. It’s just a massively worse experience than tmux + a real text editor. Plus the server is a real pig and they often end up running half a dozen server instances because the server doesn’t terminate properly. And then, half the updates break and they lose an hour because they don’t know how to ssh into the host with a real ssh client and clean up the busted vscode server.
I'm not sure what features VSCode offers exactly, but I find sshfs works quite well for remote editing of a bunch of stuff. I think it should basically be the same as VSCode?
One fewer tool to think about. It remembers the connections and instantly gives you access to a full-fledged editor on a remote server. It just works, and I don’t think about it twice, which is a huge win in my book. I’m good with the trade-offs.
Absolutely. It worked fine before vscode existed and it will work fine when the next hot button IDE comes along. I'm not sure why people keep reinventing the wheel and creating themselves new problems!
What did we learn? That remote code execution is a thing? That misplaced trust in [dev] tools is often regretted? That modern software design is ass? If you've been paying attention, all of that has been obvious.
SSH is a solution from the 90s. It's Telnet with a couple extra features bolted on, and despite being called the "secure" shell, is literally less secure than Telnet+TLS would be. There's so much stupid-yet-glorified bullshit implemented over SSH, because lazy assholes had a tunnel into a server with a user session, and decided "why implement a whole extra protocol for network transport and secure connections for my application? I already have an authenticated user session, in this very specific protocol, computing design, and network architecture. who care if nobody else on the planet but computer nerds have used something like this since 2000? works for me!" Discard the concepts learned from distributed OSes, ignore all the advanced AuthN+Z that had been developed, embrace the shittiest, easiest thing you can.
These "SSH agents" aren't bananas. We refused to get off our stupid lazy asses to build the right tool for the job. So we had to keep shoving more shit into pre-existing tools that were never designed for what we decided to do with it. We have no right to pretend we're surprised.
This is the world we have made. All of you, and me too. Either with your labor or your silent compliance. If it's not SSH, it's politics, commerce, school, and everything else. We live every day in the shit-pile we carved for ourselves, and every day that we don't do something about it, we throw another shovel on the pile. You don't get to sit there shoveling shit and pretend this is a fucking surprise, or crazy. You're holding a shovel.
It's not the developers who did this, it's the "network security" people. If you block all outgoing ports except for HTTPS and ssh, everything is henceforth going to be tunneled over HTTPS or ssh.
This is also the reason why, if you allow outgoing HTTPS connections, you should as a general rule be allowing all outgoing connections except for SMTP. Because actually malicious traffic is just going to be tunneled over HTTPS anyway and all you're doing is inhibiting the deployment of any new protocols that don't take on the complexity and inefficiency of the tunnel.
To the contrary, i find the ssh keypair authentication (and certificates) the best kind of authentication i know. And it integrates with FIDO2 without prior set up.
I wish web logins would be more like what SSH does.
Man, I'm getting hard tinfoil conspiracy vibes from this one. Not everything in life is done with malicious intent. Most of the time it's humans trying to do the best they can come up with under pressure.
Maybe go touch some grass once in a while, it's healthy for the soul.
Also, please make a suggestion on how to build a better SSH protocol. Complaints are kind of useless without constructive criticism.
1) I never said it was malicious, 2) humans do not try to do their best (have you looked around lately?), 3) complaints are entirely useful regardless of constructive criticism, they are literally user feedback
Agreed, because VSCode does not provide an SSH Agent but does communicate with your local one (their own version of ForwardAgent, complete with the security implications that carries). And the way it does that breaks a popular macOS SSH agent: https://github.com/maxgoedjen/secretive/issues/543
Oh shoot, secretive is one of the few ways I knew of to keep track of each time an SSH key was actually used to auth something. I love using Agent Forwarding but the idea of anybody with root being able to arbitrarily auth stuff gave me a (perhaps unreasonable) feeling of insecurity.
> I would be a little nervous about letting people VSCode-remote-edit stuff on dev servers, and apoplectic if that happened during an incident on something in production.
Totally agree that using vscode remote on production servers is insane.
Everything else described as "bananas" sounds like expected functionality.
I think the use case is mostly dev servers, either remote machines running in the cloud or docker containers running locally. I don't use VSCode much, but the one thing I do use it for is when I need to do something linux-specific. I spin up a linux VM on my MacBook and use VSCode to remote into the VM.
All you need is a well-defined .devcontainer file.
Debugging, extensions, collaborative coding, dependant services, OS libraries, as much RAM as you need (as opposed to what you have), specific NodeJS Versions — all with a single click.
It's dev-environment-as-code and you can version control it.
Readit News
I promise, I thought about this less than you have. It's a thing we were tinkering with, and I wrote about it. That's all.
> A compromised remote could use the VS Code Remote connection to execute code on your local machine.
I feel like that "security note" should have a CVE number next to it.
Glad to see one of my favorite blogs is still active. I was starting to get a little worried.
The first two blogs currently listed, McCord-Valim’s FLAME-Livebook-GPU and your post which has the word “murid”, truly captures the psychological arc of a developer.
Thanks!
https://www.youtube.com/watch?v=DN1Foddvez8
Although maybe the solution is to download a docker container on the remote (directly if it has access to dockehub or over an ssh connection if it doesn't) then run a docker container mounting the remote directory and connect to it over ssh.
The problem with only syncing files from the subdirectory is that you also want to be able to run &debug on the remote started by vscode. So plugins also need access to remote or to run remotely for some code observation that may be prohibitively expensive if run locally(pre sync of whole subdir)
This is, indeed, the way.
Dead Comment
Even just from the SSH extension - the workspace URIs have two formats: essentially just the hostname and hex-encoded JSON documents. The latter case happens when additional info is needed, e.g. specific username, or... The hostname includes an uppercase letter.
Which is actually necessary because when they're saved to recent workspaces they're lowercased for whatever reason.
The SSH connections also support configuring extensions that are to be installed on the server, but don't go too crazy with it or you won't be able to connect to Windows hosts, since they're passing them ass command line arguments via CMD, which has a 8191 character limit (they're using CMD to call... PowerShell...).
The most unfortunate few lines. [0]
I wish Microsoft would hire me for a couple months to do nothing but sit in a corner by myself untangling this mess.
[0] https://github.com/microsoft/vscode/blob/6dbde2a3ed308f88164...
How is this a security problem? Is it because someone on the same network as the remote machine but without SSH access can connect the port that is forwarded over SSH?
As a user, I quite like how well VSCode's SSH system works.
VSCode is installing a remote agent on the target machine that happens to use ssh as its transport protocol, and offers to share that transport with the user.
Is this a problem? Not if it only does things you want it to do. However any agent based system exposing an arbitrary API is suddenly a much bigger attack and risk surface area than the well trod (and still fraught) path of emulating a terminal over ssh.
I can see how this increases local (to the remote system) attack surface, but as long as the agent has the same OS privileges as the user logged in over SSH, what extra remote risk does this introduce?
But then I see the clusterfucks productive people are doing and it looks like I'm onto something
I can see how writing a custom agent that provides remote access to privileged API's is a bad idea but bash isn't exactly the most secure piece of software in the world.
The remote development workflow is intended and should be used for dedicated disposable development machines.
That's the responsibility of the operator and it is not for the tool to take care of.
> The agent runs over port-forwarded SSH. It establishes a WebSockets connection back to your running VSCode front-end. The underlying protocol on that connection can:
- Wander around the filesystem - Edit arbitrary files - Launch its own shell PTY processes - Persist itself
When you ssh into a remote server as a client, afaik that server cannot execute arbitrary code on the client. At a minimum, the client would have to explicitly take action for that to happen.
Deleted Comment
...assuming you have X11 forwarding disabled and/or don't have X11 server running on the same system that your client is running on.
Dead Comment
But it is a security problem in the same way that “curl | bash” is a security problem. An even closer equivalence might be something like “curl | bash” in your bashrc.
The agent is networked and running all the time so a hole on the dev server’s firewall is now a hole in your laptop’s firewall.
Of course. The problem is that a 3rd party agent can now use your permission to do what it wants, and you will be none the wiser.
Only technically. There are lots of situations where people (or robotic alarm bells) will be miffed if you instal your custom sw stacks, remote shells, "i don't know what these binaries are" etc on some servers you're supposedly just editing config files on.
Deleted Comment
I've tried a few things to fix this: 1., I set a motd on our class servers asking the students not to use the VSCode remote server plugin. 2. I ran `ncdu /home` in front of the class and demonstrated how, without exception, everyone with >100MB of disk usage on the class server was a VSCode user. 3. I set a user process limit of 45, because the VSCode RAT (somehow) uses ~50 Node processes. When students inevitably ignored the motd and the in-class warning, they hit the process cap and had to ask us to kill their processes in order to get back onto the system. 4. I replaced the process limit with a script that kills all the .vscode-server RATs every 10 seconds.
The smart kids (whom I’m sure look like they are learning that precious openssh client) are doing their assignment locally or on a free tier VPS with VS Code and scping the thing over when it’s done.
They’re also smart enough to learn openssh when they need it IRL.
cyberduck - https://cyberduck.io/
Every day we stray further from God.
VSCode doesn't show the motd :/
https://news.ycombinator.com/newsguidelines.html
The agent gets out of my way and lets me work in peace. It almost feels like I'm working from my local machine—a huge win in my book.
This could be a security risk, but the development experience is unmatched. I couldn't care less which other editor VSCode is killing, as I mostly don't care about my tools as long as they get out of the way and let me do my job.
How does it execute the exact version of clangd that is running on the remote not my local laptop, desktop, or windows machine I’m using to remote in?
How well does it search through files on a slower connection when it has to download every byte of every file?
Deleted Comment
VSCode does have a thing where it's like do you trust the authors of this project. Not sure what it does because I've never had to use it. From StackOverflow[1]:
>If you select No, I don't trust the authors, Visual Studio Code will open the workspace in 'restricted mode'. This is the default for all new workspaces. It lets you safely browse through code but disables some editor feature, including debugging, tasks, and many extensions. However, keep in mind that 'restricted mode' is all you need for many use cases.
Actually if restricted mode[2] is any good, vscode might be better at security than most other editors/IDEs.
[1]:https://stackoverflow.com/a/67914669/11422647 [2]:https://code.visualstudio.com/docs/editor/workspaces/workspa...
I'm not sure what features VSCode offers exactly, but I find sshfs works quite well for remote editing of a bunch of stuff. I think it should basically be the same as VSCode?
Emacs' TRAMP is pretty crap but still more reliable and user-friendly than the hot mess of VSCode remote editing.
Okay. I have been using VSCode remote session since its inception and it works well for my case.
Also, Vim/Emacs aren’t my cup of tea, so it’s not like that I have many options.
Students need to focus on fundamentals, not the fanciest automatic whiz-bang tools available. Why we learn arithmetic before allowing calculators.
SSH is a solution from the 90s. It's Telnet with a couple extra features bolted on, and despite being called the "secure" shell, is literally less secure than Telnet+TLS would be. There's so much stupid-yet-glorified bullshit implemented over SSH, because lazy assholes had a tunnel into a server with a user session, and decided "why implement a whole extra protocol for network transport and secure connections for my application? I already have an authenticated user session, in this very specific protocol, computing design, and network architecture. who care if nobody else on the planet but computer nerds have used something like this since 2000? works for me!" Discard the concepts learned from distributed OSes, ignore all the advanced AuthN+Z that had been developed, embrace the shittiest, easiest thing you can.
These "SSH agents" aren't bananas. We refused to get off our stupid lazy asses to build the right tool for the job. So we had to keep shoving more shit into pre-existing tools that were never designed for what we decided to do with it. We have no right to pretend we're surprised.
This is the world we have made. All of you, and me too. Either with your labor or your silent compliance. If it's not SSH, it's politics, commerce, school, and everything else. We live every day in the shit-pile we carved for ourselves, and every day that we don't do something about it, we throw another shovel on the pile. You don't get to sit there shoveling shit and pretend this is a fucking surprise, or crazy. You're holding a shovel.
This is also the reason why, if you allow outgoing HTTPS connections, you should as a general rule be allowing all outgoing connections except for SMTP. Because actually malicious traffic is just going to be tunneled over HTTPS anyway and all you're doing is inhibiting the deployment of any new protocols that don't take on the complexity and inefficiency of the tunnel.
I wish web logins would be more like what SSH does.
Maybe go touch some grass once in a while, it's healthy for the soul.
Also, please make a suggestion on how to build a better SSH protocol. Complaints are kind of useless without constructive criticism.
Totally agree that using vscode remote on production servers is insane.
Everything else described as "bananas" sounds like expected functionality.
All you need is a well-defined .devcontainer file.
Debugging, extensions, collaborative coding, dependant services, OS libraries, as much RAM as you need (as opposed to what you have), specific NodeJS Versions — all with a single click. It's dev-environment-as-code and you can version control it.