finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt.
Now if only someone could do the same for shell scripts. Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. Right now it’s still curl | bash and hope for the best, or a README with 12 manual steps and three missing dependencies.
Sure, there’s Nix... if you’ve already transcended time, space, and the Nix manual. Docker? Great, if downloading a Linux distro to run sed sounds reasonable.
There’s got to be a middle ground simple, declarative, and built for humans.
Nix is overkill for any of the things it can do. Writing a simple portable script is no exception.
But: it’s the same skill set for every one of those things. This is why it’s an investment worth making IMO. If you’re only going to ever use it for one single thing, it’s not worth it. But once you’ve learned it you’ll be able to leverage it everywhere.
Python scripts with or without dependencies, uv or no uv (through the excellent uv2nix which I can’t plug enough, no affiliation), bash scripts with any dependencies you want, etc. suddenly it’s your choice and you can actually choose the right tool for the job.
Not trying to derail the thread but it feels germane in this context. All these little packaging problems go away with Nix, and are replaced by one single giant problem XD
I don't think nix is that hard for this particular use case. Installing nix on other distros is pretty easy, and once it's installed you just do something like this
I will say this with a whole heart. My arch linux broke and I wanted to try out nix.
The most shocking part about nix is the nix-shell (I know I can use it in other distros but hear me out once), its literally so cool to install projects for one off.
Want to record a desktop? Its one of those tasks that for me I do just quite infrequently and I don't like how in arch, I had to update my system with obs as a dependency always or I had to uninstall it. Ephemerality was a concept that I was looking for before nix since I always like to try out new software/keep my home system kind of minimalist-ish Cool. nix-shell -p obs-studio & obs and you got this.
honestly, I like a lot of things about nix tbh. I still haven't gone too much into the flake sides of things and just use it imperatively sadly but I found out that nix builds are sandboxed so I found a unique idea of using it as a sandbox to run code on reddit and I think I am going to do something cool with it. (building something like codapi , codapi's creator is kinda cool if you are reading this mate, I'd love talking to ya)
And I personally almost feel as if some software could truly be made plug n play (like imagine hetzner having nix os machines (currently I have heard that its support is finnicky) but then somehow a way to get hetzner nix os machines and then I almost feel as if we can get something really really close to digital ocean droplets/ plug n play without any isolation that docker provides because I guess docker has its own usecases but I almost feel as if managing docker stuff is kinda harder than nix stuff but feel free to correct me as I am just saying what I am feelin using nix.
I also wish if something like functional lua (does fxn lua exist??) -> nix transpiler because I'd like to write lua instead of nix to manage my system but I guess nix is fine too!
> Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages.
IMO it should stay that way, because any script that needs those things is way past the point where shell is a reasonable choice. Shell scripts should be small, 20 lines or so. The language just plain sucks too much to make it worth using for anything bigger.
My rule of thumb is that as soon as I write a conditional, it's time to upgrade bash to Python/Node/etc. I shouldn't have to search for the nuances of `if` statements every time I need to write them.
When you solve the dependency management issue for shell scripts, you can also use newer language features because you can ship a newer interpreter the same way you ship whatever external dependencies you have. You don't have to limit yourself to what is POSIX, etc. Depending on how you solve it, you may even be able to switch to a newer shell with a nicer language. (And doing so may solve it for you; since PowerShell, newer shells often come with a dependency management layer.)
> any script that needs those things
It's not really a matter of needing those things, necessarily. Once you have them, you're welcome to write scripts in a cleaner, more convenient way. For instance, all of my shell scripts used by colleagues at work just use GNU coreutils regardless of what platform they're on. Instead of worrying about differences in how sed behaves with certain flags, on different platforms, I simply write everything for GNU sed and it Just Works™. Do those scripts need such a thing? Not necessarily. Is it nicer to write free of constraints like that? Yes!
Same thing for just choosing commands with nicer interfaces, or more unified syntax... Use p7zip for handling all your archives so there's only one interface to think about. Make heavy use of `jq` (a great language) for dealing with structured data. Don't worry about reading input from a file and then writing back to it in the same pipeline; just throw in `sponge` from moreutils.
> The language just plain sucks too much
There really isn't anything better for invoking external programs. Everything else is way clunkier. Maybe that's okay, but when I've rewritten large-ish shell scripts in other languages, I often found myself annoyed with the new language. What used to be a 20-line shell script can easily end up being 400 lines in a "real" language.
I kind of agree with you, of course. POSIX-ish shells have too much syntax and at the same time not enough power. But what I really want is a better shell language, not to use some interpreted non-shell language in their place.
I have a couple of projects consisting of around >1k lines of Bash. :) Not to bloat, but it is pretty easy to read and maintain. It is complete as well. I tested all of its functionalities and it just works(tm). Were it another language, it may have been more than just around 1k LOC, however, or more difficult to maintain. I call some external programs a lot, so I stick'd to a shell script.
I simply do not write shell scripts that use or reference binaries/libraries that are no pre-installed on the target OS (which is the correct target, writing shell scripts for portability is silly).
There is no package manager that is going to make a shell script I write for macOS work on Linux if that script uses commands that only exist on macOS.
That's a shame as I got to a monk-level python jujitsu. I can fix any problem, you name it, https nightmare, brew version vs pyenv, virtualenv shenanigans. Now all this knowledge is a bad investment of time.
Knowing the Python packaging ecosystem, uv could very well be replaced by something else. It feels different this time, but we won't know for a while yet.
I'm unable to resist responding that clearly the solution is to run Nix in Docker as your shell since packaging, dependency management, and reproducibility will be at theoretical maximum.
For the specific case of solving shell script dependencies, Nix is actually very straightforward. Packaging a script is a writeShellApplication call and calling it is a `nix run`.
I guess the issue is just that nobody has documented how to do that one specific thing so you can only learn this technique by trying to learn Nix as a whole.
So perhaps the thing you're envisaging could just be a wrapper for this Nix logic.
> finally feels like Python scripts can Just Work™ without a virtualenv scavenger hunt.
Hmm, last time I checked, uv installs into ~/.local/share/uv/python/cpython-3.xx and can not be installed globally e.g. inside a minimal docker without any other python.
Like the author, I find myself going more for cross-platform Python one-offs and personal scripts for both work and home and ditching Go. I just wish Python typechecking weren't the shitshow it is. Looking forward to ty, pyrefly, etc. to improve the situation a bit
Speed is one thing, the type system itself is another thing, you are basically guaranteed to hit like 5-10 issues with python's weird type system before you start grasping some of the oddities
I wouldn't describe Python type checking as a shit-show. pyright is pretty much perfect. One nit against it perhaps is that it doesn't support non-standard typing constructs like mypy does (for Django etc). That's an intentional decision on the maintainer's part. And I'm glad he made that decision because that spurned efforts to make the standard typing constructs more expressive.
I'm also looking forward to the maturity of Rust-based type checkers, but solely because one can almost always benefit from an order of magnitude improvement in speed of type checking, not because Python type-checking is a "shit show".
I do grant you that for outsiders, the fact that the type checker from the Python organization itself is actually a second rate type checker (except for when one uses Django, etc, and then it becomes first-rate) is confusing.
I've never particularly liked go for cross platform code anyway. I've always found it pretty tightly wedded to Unix. Python has its fair share of issues on windows aswell though, I've been stuck debugging weird .DLL issues with libraries for far too long in my life.
Strangely, I've found myself building personal cross platform apps in game engines because of that.
I do hope the community will converge on one type checker like ty. The fact that multiple type checkers exist is really hindering to the language as a whole.
uv has been fantastic to use for little side projects. Combining uv run with `uv tool run` AKA `uvx` means one can fetch, install within a VM, and execute Python scripts from Github super easily. No git clone, no venv creation + entry + pip install.
And uv is fast — I mean REALLY fast. Fast to the point of suspecting something went wrong and silently errored, when it fact it did just what I wanted but 10x faster than pip.
It (and especially its docs) are a little rough around the edges, but it's bold enough and good enough I'm willing to use it nonetheless.
I know there are real reasons for slow Python startup time, with every new import having to examine swaths of filesystem paths to resolve itself, but it really is a noticeable breath of fresh air working with tools implemented in Go or Rust that have sub-ms startup.
I agree uv is amazing, but it's not a virtual machine, it's a virtual environment. It runs the scripts on top of your OS without any hardware virtualization. The virtual environment only isolates the Python dependencies.
not sure if OP will see this, but how does uv or uvx forgo `git clone` ? You still need to clone whatever you're trying to run, or am I missing something.
Is there a reason you didn’t explicitly pull in mkdocs as a dependency in that invocation? I guess uv will expose it/let you run it anyways due to the fact that it’s required by everything else you did specify.
Very nice, I believe Rust is doing something similar too which is where I initially learned of this idea of single-file shell-type scripts in other languages (with dependency management included, which is how it differs from existing ways of writing single-file scripts in e.g. scripting languages) [0].
Hopefully more languages follow suit on this pattern as it can be extremely useful for many cases, such as passing gists around, writing small programs which might otherwise be written in shell scripts, etc.
> Before this I used to prefer Go for one-off scripts because it was easy to create a self-contained binary executable.
I still do because:
- Go gives me a single binary
- Dependencies are statically linked
- I don’t need any third-party libs in most scenarios
- Many of my scripts make network calls, and Go has a better stdlib for HTTP/RPC/Socket work
- Better tooling (built-in formatter, no need for pytest, go vet is handy)
- Easy concurrency. Most of my scripts don’t need it, but when they do, it’s easier since I don’t have to fiddle with colored functions, external libs, or, worse, threads.
That said, uv is a great improvement over the previous status quo. But I don’t write Python scripts for reasons that go beyond just tooling. And since it’s not a standard tool, I worry that more things like this will come along and try to “improve” everything. Already scarred and tired in that area thanks to the JS ecosystem. So I tend to prefer stable, reliable, and boring tools over everything else. Right now, Go does that well enough for my scripting needs.
I needed to process a 2 GB xml file the other day. While my Python script was chugging away, I had Claude translate it to Go. The vibe-coded Go program then processed the file before my original Python script terminated. That was the first time I ever touched Go, but it certainly won't be the last.
I still use both Go and Python. But Python gives me access to a lot more libraries that do useful stuff. For example the YouTube transcript example I wrote about in the article was only possible in Python because afaik Go doesn't have a decent library for transcript extraction.
So far I've only run into one minor ergonomic issue when using `uv run --script` with embedded metadata which is that sometimes I want to test changes to the script via the Python REPL, but that's a bit harder to do since you have to run something like:
$ uv run --python=3.13 --with-requirements <(uv export --script script.py) -- python
>>> from script import X
I'd love if there were something more ergonomic like:
$ uv run --with-script script.py python
Edit: this is better:
$ "$(uv python find --script script.py)"
>>> from script import X
That fires up the correct python and venv for the script. You probably have to run the script once to create it.
You can make `--interactive` or whatever you want a CLI flag from the script. I often make these small Typer CLIs with something like that (or in this case, in another dev script like this, I have `--sql` for entering a DuckDB SQL repl)
I’ve been a python dev for nearly a decade and never once thought dep management was a problem.
If I’ve ever had to run a “script” in any type of deployed ENV it’s always been done in that ENVs python shell .
So I still don’t see what the fuss is about?
I work on a massive python code base and the only benefit I’ve seen from moving to UV is it has sped up dep installation which has had positive impact on local and CI setup times.
I guess this is why people need to get out of this “Python dev” or “JS dev” mindset and try other languages to see why those coming to Python complain so much about dependency management.
People complain because the experience is less confusing in many other languages. Think Go, Rust, or even JS. All the tooling chaos and virtual environment jujitsu are real deterrents for newcomers. And it’s not just beginners complaining about Python tooling. Industry veterans like Armin Ronacher do that all the time.
uv is a great step in the right direction, but the issue is that as long as the basic tooling isn’t built into the language binary, like Go’s tools or Rust’s Cargo, more tools will pop up and fragment the space even further.
Confusing is underselling it. That implies that Python dependency management is working fine, it's just complex. But it's not working fine: there's no such thing as lock files, which makes reproducible installs a gamble and not a given. For small scripts this is probably "okay", but if you're working in a team or want to deploy something on a server, then it's absolutely not fine because you want deterministic builds and that's simply impossible without a decent package manager.
Tools like uv solve the "it works on my machine" problem. And it's also incredibly fast.
My view is I’m an engineer first and foremost and I use the tools which are best for the task at hand. That also means what’s best for the business in terms of others working on the project, this has meant python with some sort of framework.
People have suggested using other languages that might be faster but the business always choices what’s best for everyone to work with.
> it’s always been done in that ENVs python shell .
What if you don't have an environment set up? I'm admittedly not a python expert by any means but that's always been a pain point for me. uvx makes that so much easier.
I wrote PHP/JS/Java before Python. Been doing Python for nearly a decade too, and like 4dregress haven't had the need to worry much about dep management. JS and PHP had all sorts of issues, Maven & Gradle are still the ones that gave me less trouble. With Python I found that most issues could be fixed by finding the PEP that implemented what I needed, and by trying to come up with a simple workflow & packaging strategy.
Nowadays I normally use `python venv/bin/<some-executable>`, or `conda run -n <some-env> <some-executable>`, or packaged it in a Singularity container. And even though I hear a lot of good things about uv, given that my job uses public money for research, we try to use open source and standards as much as possible. My understanding is that uv is still backed by a company, and at least when I checked it some time ago (in peps discussions & GH issues) they were no implementing the PEPs that I needed -- even if they did, we would probably still stay with simple pip/setuptools to avoid having to use research budget to update our build if the company ever changed its business model (e.g. what anaconda did some months/year? ago).
Digressing: the Singularity container is useful for research & HPC too, as it creates a single archive, which is faster to load on distributed filesystems like the two I work on (GPFS & LustreFS) instead of loading many small files over network.
I've been a python dev for nearly 3 decades and feel that uv is removing a lot of the rough edges around dependency management. So maybe "problem" is the wrong word; I've been able to solve dependency management issues usually without too much trouble, I have also spent a significant amount of time dealing with them. For close to a decade I was managing other peoples Python environments on production systems, and that was a big mess, especially with trying to ensure that they stayed updated and secure.
If you don't see what the fuss is about, I'm happy for you. Sounds like you're living in a fairly isolated environment. But I can assure you that uv is worth a lot of fussing about, it's making a lot of our lives a lot easier.
Virtual environments alone are not enough. They don't guarantee deterministic builds. What do you do to ensure that your production environment runs the same code as your local dev environment? How do you solve that problem without dependency managers like uv or poetry?
Readit News
Now if only someone could do the same for shell scripts. Packaging, dependency management, and reproducibility in shell land are still stuck in the Stone Ages. Right now it’s still curl | bash and hope for the best, or a README with 12 manual steps and three missing dependencies.
Sure, there’s Nix... if you’ve already transcended time, space, and the Nix manual. Docker? Great, if downloading a Linux distro to run sed sounds reasonable.
There’s got to be a middle ground simple, declarative, and built for humans.
But: it’s the same skill set for every one of those things. This is why it’s an investment worth making IMO. If you’re only going to ever use it for one single thing, it’s not worth it. But once you’ve learned it you’ll be able to leverage it everywhere.
Python scripts with or without dependencies, uv or no uv (through the excellent uv2nix which I can’t plug enough, no affiliation), bash scripts with any dependencies you want, etc. suddenly it’s your choice and you can actually choose the right tool for the job.
Not trying to derail the thread but it feels germane in this context. All these little packaging problems go away with Nix, and are replaced by one single giant problem XD
ChatGPT writes pretty good nix now. You can simply paste any errors in and it will fix it.
If you do care, then welcome to learning about niv, flakes, etc.
[0]: admittedly 3 years ago or so.
The most shocking part about nix is the nix-shell (I know I can use it in other distros but hear me out once), its literally so cool to install projects for one off.
Want to record a desktop? Its one of those tasks that for me I do just quite infrequently and I don't like how in arch, I had to update my system with obs as a dependency always or I had to uninstall it. Ephemerality was a concept that I was looking for before nix since I always like to try out new software/keep my home system kind of minimalist-ish Cool. nix-shell -p obs-studio & obs and you got this.
honestly, I like a lot of things about nix tbh. I still haven't gone too much into the flake sides of things and just use it imperatively sadly but I found out that nix builds are sandboxed so I found a unique idea of using it as a sandbox to run code on reddit and I think I am going to do something cool with it. (building something like codapi , codapi's creator is kinda cool if you are reading this mate, I'd love talking to ya)
And I personally almost feel as if some software could truly be made plug n play (like imagine hetzner having nix os machines (currently I have heard that its support is finnicky) but then somehow a way to get hetzner nix os machines and then I almost feel as if we can get something really really close to digital ocean droplets/ plug n play without any isolation that docker provides because I guess docker has its own usecases but I almost feel as if managing docker stuff is kinda harder than nix stuff but feel free to correct me as I am just saying what I am feelin using nix.
I also wish if something like functional lua (does fxn lua exist??) -> nix transpiler because I'd like to write lua instead of nix to manage my system but I guess nix is fine too!
IMO it should stay that way, because any script that needs those things is way past the point where shell is a reasonable choice. Shell scripts should be small, 20 lines or so. The language just plain sucks too much to make it worth using for anything bigger.
> any script that needs those things
It's not really a matter of needing those things, necessarily. Once you have them, you're welcome to write scripts in a cleaner, more convenient way. For instance, all of my shell scripts used by colleagues at work just use GNU coreutils regardless of what platform they're on. Instead of worrying about differences in how sed behaves with certain flags, on different platforms, I simply write everything for GNU sed and it Just Works™. Do those scripts need such a thing? Not necessarily. Is it nicer to write free of constraints like that? Yes!
Same thing for just choosing commands with nicer interfaces, or more unified syntax... Use p7zip for handling all your archives so there's only one interface to think about. Make heavy use of `jq` (a great language) for dealing with structured data. Don't worry about reading input from a file and then writing back to it in the same pipeline; just throw in `sponge` from moreutils.
> The language just plain sucks too much
There really isn't anything better for invoking external programs. Everything else is way clunkier. Maybe that's okay, but when I've rewritten large-ish shell scripts in other languages, I often found myself annoyed with the new language. What used to be a 20-line shell script can easily end up being 400 lines in a "real" language.
I kind of agree with you, of course. POSIX-ish shells have too much syntax and at the same time not enough power. But what I really want is a better shell language, not to use some interpreted non-shell language in their place.
There is no package manager that is going to make a shell script I write for macOS work on Linux if that script uses commands that only exist on macOS.
If you're allowed to install any deps go with uv, it'll do the rest.
I'm also kinda in love with https://babashka.org/ check it out if you like Clojure.
We use it at $work to manage dev envs and its much easier than Docker and Nix.
It also installs things in parallel, which is a huge bonus over plain Dockerfiles
Knowing the Python packaging ecosystem, uv could very well be replaced by something else. It feels different this time, but we won't know for a while yet.
I guess the issue is just that nobody has documented how to do that one specific thing so you can only learn this technique by trying to learn Nix as a whole.
So perhaps the thing you're envisaging could just be a wrapper for this Nix logic.
There's a reason why distroless images exist. :)
Hmm, last time I checked, uv installs into ~/.local/share/uv/python/cpython-3.xx and can not be installed globally e.g. inside a minimal docker without any other python.
So basically it still runs in a venv.
https://simonwillison.net/2024/Dec/19/one-shot-python-tools/
And there was a March discussion of a different blog post:
https://news.ycombinator.com/item?id=43500124
I hope this stays on the front page for a while to help publicize it.
at the end of article it shows an mcp to fetch youtube subs. I've made a similar one using simonw's llm as a fragment, if you find it useful.
https://github.com/redraw/llm-fragments-youtubeI'm also looking forward to the maturity of Rust-based type checkers, but solely because one can almost always benefit from an order of magnitude improvement in speed of type checking, not because Python type-checking is a "shit show".
I do grant you that for outsiders, the fact that the type checker from the Python organization itself is actually a second rate type checker (except for when one uses Django, etc, and then it becomes first-rate) is confusing.
Strangely, I've found myself building personal cross platform apps in game engines because of that.
And uv is fast — I mean REALLY fast. Fast to the point of suspecting something went wrong and silently errored, when it fact it did just what I wanted but 10x faster than pip.
It (and especially its docs) are a little rough around the edges, but it's bold enough and good enough I'm willing to use it nonetheless.
But that ship has sailed for me and I'm a uv convert.
Thought I was the only one thinking this. Got to open an issue, I think it would be nice to have some more examples showcasing different use cases.
Hopefully more languages follow suit on this pattern as it can be extremely useful for many cases, such as passing gists around, writing small programs which might otherwise be written in shell scripts, etc.
[0] https://rust-lang.github.io/rfcs/3424-cargo-script.html
I still do because:
- Go gives me a single binary
- Dependencies are statically linked
- I don’t need any third-party libs in most scenarios
- Many of my scripts make network calls, and Go has a better stdlib for HTTP/RPC/Socket work
- Better tooling (built-in formatter, no need for pytest, go vet is handy)
- Easy concurrency. Most of my scripts don’t need it, but when they do, it’s easier since I don’t have to fiddle with colored functions, external libs, or, worse, threads.
That said, uv is a great improvement over the previous status quo. But I don’t write Python scripts for reasons that go beyond just tooling. And since it’s not a standard tool, I worry that more things like this will come along and try to “improve” everything. Already scarred and tired in that area thanks to the JS ecosystem. So I tend to prefer stable, reliable, and boring tools over everything else. Right now, Go does that well enough for my scripting needs.
I still use both Go and Python. But Python gives me access to a lot more libraries that do useful stuff. For example the YouTube transcript example I wrote about in the article was only possible in Python because afaik Go doesn't have a decent library for transcript extraction.
Here's how it's relevant :)
You can make `--interactive` or whatever you want a CLI flag from the script. I often make these small Typer CLIs with something like that (or in this case, in another dev script like this, I have `--sql` for entering a DuckDB SQL repl)
If I’ve ever had to run a “script” in any type of deployed ENV it’s always been done in that ENVs python shell .
So I still don’t see what the fuss is about?
I work on a massive python code base and the only benefit I’ve seen from moving to UV is it has sped up dep installation which has had positive impact on local and CI setup times.
"missing x..."
pip install x
run script
"missing y..."
pip install y
> y not found
google y to find package name
pip install ypackage
> conflict with other package
realize I forgot a venv and have contaminated my system python
check pip help output to remember how to uninstall a package
clean up system python
create venv at cwd
start over
...
</end of time>
People complain because the experience is less confusing in many other languages. Think Go, Rust, or even JS. All the tooling chaos and virtual environment jujitsu are real deterrents for newcomers. And it’s not just beginners complaining about Python tooling. Industry veterans like Armin Ronacher do that all the time.
uv is a great step in the right direction, but the issue is that as long as the basic tooling isn’t built into the language binary, like Go’s tools or Rust’s Cargo, more tools will pop up and fragment the space even further.
Tools like uv solve the "it works on my machine" problem. And it's also incredibly fast.
People have suggested using other languages that might be faster but the business always choices what’s best for everyone to work with.
What if you don't have an environment set up? I'm admittedly not a python expert by any means but that's always been a pain point for me. uvx makes that so much easier.
Nowadays I normally use `python venv/bin/<some-executable>`, or `conda run -n <some-env> <some-executable>`, or packaged it in a Singularity container. And even though I hear a lot of good things about uv, given that my job uses public money for research, we try to use open source and standards as much as possible. My understanding is that uv is still backed by a company, and at least when I checked it some time ago (in peps discussions & GH issues) they were no implementing the PEPs that I needed -- even if they did, we would probably still stay with simple pip/setuptools to avoid having to use research budget to update our build if the company ever changed its business model (e.g. what anaconda did some months/year? ago).
Digressing: the Singularity container is useful for research & HPC too, as it creates a single archive, which is faster to load on distributed filesystems like the two I work on (GPFS & LustreFS) instead of loading many small files over network.
python3 -m venv venv
Activate the virtual environment:
source venv/bin/activate
Deactivate the virtual environment:
deactivate
If you don't see what the fuss is about, I'm happy for you. Sounds like you're living in a fairly isolated environment. But I can assure you that uv is worth a lot of fussing about, it's making a lot of our lives a lot easier.