Used this at my last company. Super painless and makes setup a breeze. Current company uses Docker and I hate it. Every company I've been at loves to throw everything into a container. Even if you setup your dev configs to maintain hot reloading, it's always slower. Node package management is also a pain no matter how you implement it with Docker it seems.
I also hate the Docker tagline that it "eliminates 'works on my machine' issues". I believe a tool like asdf would also achieve this (correct me if I'm wrong). Docker itself can go haywire depending on the machine and you're basically in hell fighting with it just to get your dev environment working. You essentially eliminate one problem in exchange for a variety of equally frustrating challenges.
I personally love docker. Almost all of it makes sense to me. It's not perfect, but very pragmatic, almost in a blunt way.
On a workstation level, people who really know their way around stuff tend to dislike docker in my experience, and for people who don't its still too difficult. One problem I think it has is that docker - still talking about workstations here - solves mostly problems on the team level, not on an individual level. But individuals tend to regard those problems as 'other peoples problems'.
Another problem is that you can work with docker just fine with barely knowing how it works, until there's a problem. In my experience, developers tend to immediately blame docker for the problem, even though its a misconfiguration of npm for example, which they would also have in a native installation.
Still, docker solves a lot of problems that aren't solved by asdf, apt or rpm, which is why it continues to be relevant and used also for workstations.
EDIT: docker on windows before wsl2 was hell too, a lot of discontent about docker that people have comes from those installs. On wsl2 it's mostly fine, if you don't attempt to mount from an ntfs file system.
In my experience as web (mostly) backend developer on Linux targeting Linux servers:
- docker is good when the deployed software runs on docker too, because dev and prod are the same and on Linux the performance hit is negligible if any. If you're on Mac or Windows it seems that it can be felt.
- docker is a good way to distribute the development environment, anecdotally much better than Vagrant or bare VMs.
- However having to always prefix commands with docker exec container is a little pain. Working in docker exec container bash is worse.
- Sharing only volumes between the host file system and docker means that sometimes I have to copy files to the shared volume instead of addressing them where they are.
- asdf doesn't have all those problems but I must be careful to pick different ports for different postgresql servers in different projects. Docker would shield those ports inside the network of the project.
All considered, I use docker when a customer uses docker, VMs when a customer uses VMs, asdf for all the other cases including my own software which I never deployed to docker so far. If I did I'd probably work in docker because I don't want surprises at deployment time.
In reality, when using Docker, Dev and Prod are almost never the same. We usually install different packages in Dev image i.e. debuggers or linters, we try to optimize the heck out of the Prod image, thus using a different build of the base image, etc.
And all this, when you are only developing on Linux for Linux.
I’m not saying this is the right way but this is very common. I also agree with the rest of your points.
I found docker to be pleasant to work with for support services (if your app needs postgres, redis, mail service, pdf converter, object storage, etc) in development and you need an easy command (docker-compose down; docker-compose up) to tell designers and product people to easily reset their environment to a sane state.
It is NOT great for having a live-reloading dev webservice running. People seem to swing for an all in one solution, but consideration is needed about what services you want in docker or not. Docker is great to have preconfigured database environments that are easy to tear down and start from scratch and works across multiple platforms. Docker is bad for local dev server.
> It is NOT great for having a live-reloading dev webservice running.
Oh my god this.
I was running a fastapi server on my m1 Mac in an arm Linux docker container. Everything else works great in this, but the live-reloading server was running at 200% cpu and ate 50% of my battery life!
If you genuinely think you have a better way to do it, propose it. Don't sit back with the terrible attitude of "my company does it this way which sucks" - why not try to make things better?
There are two possibilities - a) you are right, you know something your company doesn't, and by proposing your new plan in a constructive manner you might make life better for everyone b) you are wrong, you don't fully comprehend the surface area of the software you are working on, and your suggestion is short sighted and doesn't account for all of the requirements.
Either way it is helpful to put it out there - so long as you do it from the right place, with the right intention. Either you'll humble yourself, or you will eliminate an unnecessary dependency and make life easier for your team.
> I believe a tool like asdf would also achieve this (correct me if I'm wrong).
This is true for very simple applications, where you’re just running `npm ci` or `pip install`, but breaks down as soon as you have sophisticated dependencies with system / library requirements or dependencies (e.g. databases, caches, message brokers, queues) that are beyond the scope of asdf.
Also, since you’re probably deploying containers to production, it’s useful to have a similar environment for local development so that you know what will actually happen in production.
Best dev setup I've seen has been to use Docker compose for all of those dependencies, but just use asdf for the main application development. You get all the dependency wins of Docker for the peripherals but with the faster native local dev experience.
> Also, since you’re probably deploying containers to production, it’s useful to have a similar environment for local development so that you know what will actually happen in production.
This such a complete and utter lie and I'm surprised people in 2022 still believe it.
You do know what's happening when you run Docker on your Macbook, right? Right?
The only problems I’ve ever seen a coworker or myself encounter with docker is not knowing it well enough. Admittedly it has a strong learning curve. There are things that will not work unless you know certain underpinnings of docker, dns/networking, etc. But to say it doesn’t work and goes haywire doesn’t make sense. It’s an extremely stable piece of software.
> I also hate the Docker tagline that it "eliminates 'works on my machine' issues".
Docker basically just takes a snapshot, which hides the problem rather than fixing it: when we want to change something, like updating a dependency, we're back to the same dependency hell.
Even worse, those snapshots are often not reproducible (e.g. running things like `apt-get install -y foo`, which depends on the latest contents of third-party servers). Again, Docker tries to hide the problem by putting snapshots into a cache.
To avoid these problems, we need the discipline to do sensible things (e.g. using specific .deb packages, rather than apt-getting whatever's latest; or using something more brute-force like Nix). Yet once we do that, there's usually no point doing it with Docker at all; since those commands work perfectly well outside of a container (if we want a container to deploy, we can tar up the resulting directory)
What use case are you addressing here? Docker can be used to maintain development tools in various ways, but I see it as mostly operating on a different level from asdf.
For example, asdf can install tools on a developer's system. asdf can route invocations to the appropriate version of an installed tool.
Docker can be used to install tools in isolation by way of an image build. That image can be shipped off, so that developers can pull the image, and do not have to run an installation procedure. Docker can pull and run the appropriate version on demand upon invocation.
It's also not a binary choice. You can use asdf in a Docker container, and you could use Docker in an asdf plugin.
I've written a rust version of asdf over the last few months, both as an exercise and to fix a few issues I've been having with it - like noticeably slower tool startup times due to the shims resolving the version (my version basically works like direnv under the hood and hooks into the shell directly) and me just wanting a single "use" command which installs a plugin, downloads the requested version and sets it as active. It works very well and I've been daily driving it for a while.
Having the ecosystem of asdf plugins which are basically just shell scripts has been a huge boon. It's been a breeze to work with, and most of the plugins are well written.
Now, I've been contemplating switching to NixOS, but most version managers don't work at all with it due to dynamic linking. I absolutely love the idea of NixOS, but this has really bummed me out. I feel like the nix language is still a little clunky for general use, so as long as there is not a straightforward solution like having a tool-versions file I'm really hesitant to make the full switch.
I feel like they're just hitting the limits of a bash+shims based approach. I understand why they've done it, but in the end an implementation in a compiled language will beat that approach most of the time.
There are issues since a shell hook will only work as long as you're, well, using a (supported) shell.
However, it's surprising to hear that they've been focusing on UX. The fact that you have to manually install a plugin, then install a version, and then finally set it, without the tool suggesting any of those steps to you when you forgot one, really makes the tool annoying to use when switching versions a lot.
I don't use NixOS myself, but have Nix installed on my Mac, and it seems to provide all functionality of package or version managers I needed.
I think though it is more complex because it is a programming language that provides this functionality instead of purpose build tool like asdf.
For my needs I created a framework for development: https://github.com/takeda/nix-cde to avoid cruft of including the same things over and over in my projects.
I first started using nix thinking of it like a version manager also, but that’s not what it is, and thinking of it that way will only bring pain.
Nix is a package manager (like apt or whatever), and like package managers, it works with a set of vetted, decently verified package versions (NixOS releases). Mostly this is fine. It’s easy to use nix stable for most stuff and nix-unstable for anything where you need newer stuff. You can then configure “overlays” to the set of packages where you specify how to build particular versions where you need it.
It took me a long time to make the mental shift from nix being something like pip or npm or asdf or pyenv or whatever to thinking of it like a true linux distribution’s package manager, but it helped everything to click once I did
Thanks, that's a helpful perspective. I saw that there's also specific packages for things like node versions, so that might be an alternative to writing overlays in some cases.
I'm just concerned about packages without any support being extremely cumbersome to install, since it's hard to run regular binaries.
Thank you for sharing - I've written my own "version" of asdf in Bash too, and I'm getting to the point where I would like to write some portions in Rust.
Does qwer support per-directory, local versions of plugins? I'm finding that that's difficult to do without shims...
> noticeably slower tool startup times due to the shims resolving the version
I migrated to asdf from nvm because of nvm's very slow startup time. I haven't noticed excessive startup time on asdf yet. I wonder what's the issue with your asdf installation.
You can find it at https://github.com/happenslol/qwer . Be warned though, it's heavily lacking in documentation and polish, although all core features are working. I've just added a bit of info to the Readme, hit me up if you have any questions.
"ASDF (Another System Definition Facility) is an extensible build facility for Common Lisp software. ASDF comes bundled with all recent releases of active Common Lisp implementations as well as with quicklisp, and it is the most widely used system definition facility for Free Software Lisp libraries."
Does asdf come with a plugin for asdf & quicklisp?
would've preferred for the new guy on the block pay respect to the old guy on the block and used a different name..
One tip that I didn't know about and didn't seem to be documented at the time I was looking: if you want to add a plugin for some new project but just defer to what you were already using for everything else, you can set your `global` option to `system`. e.g.:
I have been using asdf for quite a few years, and I've always been impressed. It's honestly a breath of fresh air to only have a single set of commands to remember for node, go, ruby, python, even crystal. For node, it even respects existing .nvmrc files.
After the pain of pyenv and it's terrible compatibility with poetry, asdf has been a dream come true. It just works, every time. No fragile config. No mess.
One of the things Pyenv let's you do is to use a specific version of Python within a folder. Poetry will sometimes fail to detect it and use whatever version is used globally instead.
Rarely have I seen a tool adopted so swiftly and at such scale. It eats the role of hundreds of other language & tool specific version managers, and whatever the secret sauce is, it's nicely fast & relatively low pain. Hats off to asdf. Major major delta in devs lives from this recent-ist new more-meta contender.
I don't think I've even seen an asdf plugin that directly uses a version manager. They generally seem to either A) download prebuilt binaries or B) fallback to building from source. Either way the installs go straight to ~/.asdf/installs.
The only exception I can think of is any dependencies on libraries that may or may not be installed by the system package manager. Even then, those dependencies are often optional.
The secret sauce is a low barrier to entry. An asdf tool is just a shell script that does the work to list packages, download and install them, etc. You can make a simple tool script in an afternoon.
The low barrier to entry is also a bit of a curse too. The quality of asdf tool scripts varies wildly. Anything not maintained by their core org is a crapshoot in my experience. Many of those smaller tools don't support ARM or have lots of undocumented dependencies or quirks.
`asdf` is a great tool but I'd like to remind people that it's probably a smart idea to be agnostic to it, for the very same reasons they should've been agnostic to their previous technologies of choice that `asdf` had replaced.
I've been using this at work, sharing with my colleagues, and integrating this into our default CI/CD environment. It's awesome and has taken away the thought of "how do I install version XYZ of tool ABC here?" entirely. Just asdf what I need, done. Huge productivity boost.
Readit News
I also hate the Docker tagline that it "eliminates 'works on my machine' issues". I believe a tool like asdf would also achieve this (correct me if I'm wrong). Docker itself can go haywire depending on the machine and you're basically in hell fighting with it just to get your dev environment working. You essentially eliminate one problem in exchange for a variety of equally frustrating challenges.
On a workstation level, people who really know their way around stuff tend to dislike docker in my experience, and for people who don't its still too difficult. One problem I think it has is that docker - still talking about workstations here - solves mostly problems on the team level, not on an individual level. But individuals tend to regard those problems as 'other peoples problems'.
Another problem is that you can work with docker just fine with barely knowing how it works, until there's a problem. In my experience, developers tend to immediately blame docker for the problem, even though its a misconfiguration of npm for example, which they would also have in a native installation.
Still, docker solves a lot of problems that aren't solved by asdf, apt or rpm, which is why it continues to be relevant and used also for workstations.
EDIT: docker on windows before wsl2 was hell too, a lot of discontent about docker that people have comes from those installs. On wsl2 it's mostly fine, if you don't attempt to mount from an ntfs file system.
- docker is good when the deployed software runs on docker too, because dev and prod are the same and on Linux the performance hit is negligible if any. If you're on Mac or Windows it seems that it can be felt.
- docker is a good way to distribute the development environment, anecdotally much better than Vagrant or bare VMs.
- However having to always prefix commands with docker exec container is a little pain. Working in docker exec container bash is worse.
- Sharing only volumes between the host file system and docker means that sometimes I have to copy files to the shared volume instead of addressing them where they are.
- asdf doesn't have all those problems but I must be careful to pick different ports for different postgresql servers in different projects. Docker would shield those ports inside the network of the project.
All considered, I use docker when a customer uses docker, VMs when a customer uses VMs, asdf for all the other cases including my own software which I never deployed to docker so far. If I did I'd probably work in docker because I don't want surprises at deployment time.
I’m not saying this is the right way but this is very common. I also agree with the rest of your points.
It is NOT great for having a live-reloading dev webservice running. People seem to swing for an all in one solution, but consideration is needed about what services you want in docker or not. Docker is great to have preconfigured database environments that are easy to tear down and start from scratch and works across multiple platforms. Docker is bad for local dev server.
Oh my god this.
I was running a fastapi server on my m1 Mac in an arm Linux docker container. Everything else works great in this, but the live-reloading server was running at 200% cpu and ate 50% of my battery life!
There are two possibilities - a) you are right, you know something your company doesn't, and by proposing your new plan in a constructive manner you might make life better for everyone b) you are wrong, you don't fully comprehend the surface area of the software you are working on, and your suggestion is short sighted and doesn't account for all of the requirements.
Either way it is helpful to put it out there - so long as you do it from the right place, with the right intention. Either you'll humble yourself, or you will eliminate an unnecessary dependency and make life easier for your team.
It is totally valid to point out failures in something without knowing how to put them right, or being in a position to do so.
This is true for very simple applications, where you’re just running `npm ci` or `pip install`, but breaks down as soon as you have sophisticated dependencies with system / library requirements or dependencies (e.g. databases, caches, message brokers, queues) that are beyond the scope of asdf.
Also, since you’re probably deploying containers to production, it’s useful to have a similar environment for local development so that you know what will actually happen in production.
Citation needed I feel - I’d wager the number of containerized deployments pales in comparison to the number of binaries scp’d into a production host.
Docker is NOT about local dev, it's about deployment.
We forgot this...
This such a complete and utter lie and I'm surprised people in 2022 still believe it.
You do know what's happening when you run Docker on your Macbook, right? Right?
Deleted Comment
Docker basically just takes a snapshot, which hides the problem rather than fixing it: when we want to change something, like updating a dependency, we're back to the same dependency hell.
Even worse, those snapshots are often not reproducible (e.g. running things like `apt-get install -y foo`, which depends on the latest contents of third-party servers). Again, Docker tries to hide the problem by putting snapshots into a cache.
To avoid these problems, we need the discipline to do sensible things (e.g. using specific .deb packages, rather than apt-getting whatever's latest; or using something more brute-force like Nix). Yet once we do that, there's usually no point doing it with Docker at all; since those commands work perfectly well outside of a container (if we want a container to deploy, we can tar up the resulting directory)
I hope you don't COPY your entire system into the container at least.
For example, asdf can install tools on a developer's system. asdf can route invocations to the appropriate version of an installed tool.
Docker can be used to install tools in isolation by way of an image build. That image can be shipped off, so that developers can pull the image, and do not have to run an installation procedure. Docker can pull and run the appropriate version on demand upon invocation.
It's also not a binary choice. You can use asdf in a Docker container, and you could use Docker in an asdf plugin.
Having the ecosystem of asdf plugins which are basically just shell scripts has been a huge boon. It's been a breeze to work with, and most of the plugins are well written.
Now, I've been contemplating switching to NixOS, but most version managers don't work at all with it due to dynamic linking. I absolutely love the idea of NixOS, but this has really bummed me out. I feel like the nix language is still a little clunky for general use, so as long as there is not a straightforward solution like having a tool-versions file I'm really hesitant to make the full switch.
http://stratus3d.com/blog/2022/08/11/asdf-performance/
However, it's surprising to hear that they've been focusing on UX. The fact that you have to manually install a plugin, then install a version, and then finally set it, without the tool suggesting any of those steps to you when you forgot one, really makes the tool annoying to use when switching versions a lot.
I think though it is more complex because it is a programming language that provides this functionality instead of purpose build tool like asdf.
For my needs I created a framework for development: https://github.com/takeda/nix-cde to avoid cruft of including the same things over and over in my projects.
Nix is a package manager (like apt or whatever), and like package managers, it works with a set of vetted, decently verified package versions (NixOS releases). Mostly this is fine. It’s easy to use nix stable for most stuff and nix-unstable for anything where you need newer stuff. You can then configure “overlays” to the set of packages where you specify how to build particular versions where you need it.
It took me a long time to make the mental shift from nix being something like pip or npm or asdf or pyenv or whatever to thinking of it like a true linux distribution’s package manager, but it helped everything to click once I did
Does qwer support per-directory, local versions of plugins? I'm finding that that's difficult to do without shims...
I migrated to asdf from nvm because of nvm's very slow startup time. I haven't noticed excessive startup time on asdf yet. I wonder what's the issue with your asdf installation.
Does asdf come with a plugin for asdf & quicklisp? would've preferred for the new guy on the block pay respect to the old guy on the block and used a different name..
asdf for common lisp been around for years and used by the clergy of programming basically and still widely used today
Mostly by reusing existing version managers, which have been developed and optimised over the years. Asdf itself is a relatively thin wrapper really.
The only exception I can think of is any dependencies on libraries that may or may not be installed by the system package manager. Even then, those dependencies are often optional.
The low barrier to entry is also a bit of a curse too. The quality of asdf tool scripts varies wildly. Anything not maintained by their core org is a crapshoot in my experience. Many of those smaller tools don't support ARM or have lots of undocumented dependencies or quirks.