Ask HN: Why is Python package management still a dumpster fire?

You have to get these things right early-on, and get everyone on the same page. Once things start to get fragmented, once you have historical decisions that prevent you from improving certain aspects, etc, there's only so much you can do to after the fact to paper over the gaps.

Python's packaging story is rotten in its bones. I think at this point it's nearly impossible to fix (though many continue to try).

The way I see it, a solution would require:

- A cohesive, self-contained, easy to use, all-inclusive toolset, with a completely standardized manifest format for packages. And, for all that is holy, doesn't mess with the global environment so that we have to lie to it with virtual environments.

- That toolset would have to get overwhelming adoption by the community, to where it retroactively becomes The Python Package Manager (outside of legacy codebases, which would continue to languish). This would probably require endorsement by the Python developers, or for the tool to be so unassailably better that it's preferred without exception, or possibly both. Otherwise we'll continue to have: https://xkcd.com/927/

I want to emphasize that on the second point 50% adoption is not enough. 70% adoption is not enough. I'm talking 90%+ adoption for libraries and new projects; everything outside of it needs to be a rounding error, or we're still in fragmentation hell.

And then even in the best case- there would be a long tail for years to come of packages that haven't been ported to the new tool/manifest format, where you have to duck out to one of the other package managers to pull them in.

hardwaregeek · 4 years ago

Yeah at this point it appears to be a cultural issue. There’s been plenty of opportunity to anoint a proper package manager that doesn’t require virtual environments or install things globally by default, that uses lock files and resolves dependencies without too much fuss. If I’m not mistaken poetry and pipenv offer this. But there just isn’t the buy in. People in the Python world, including many in this thread, just seem to refuse this path and insist, proudly, that their workflow is just fine. It’s a kind of psychological curiosity that people would resist proper package management so fervently.

PaulHoule · 4 years ago

The average data scientist thinks it is OK to have a manual process to make the monthly report so long as it takes less than 1 month and so does their manager. And so does the VC that funds it.

I mean, you can’t write a command line program which is reliable and straightforward to install in blub, why would anybody need it?

melony · 4 years ago

There's also the fact that Python is old. Python is used as Linux's unofficial system scripting language the same way Macs use Ruby. I think we should mandate a SYSTEM_PYTHON variable to avoid polluting the PYTHONPATH further with brittle global installs that can't be easily upgraded or replaced.

rcarmo · 4 years ago

Erm. Brew uses Ruby, and there are historical reasons about that, but Apple never had any favorites in that race.

majou · 4 years ago

https://xkcd.com/1987/

BerislavLopac · 4 years ago

> - A cohesive, self-contained, easy to use, all-inclusive toolset,

The only complex thing about Python packaging is that is separates it tools into "frontend" and "backend" build tools. On the frontend, pip is essentially the cohesive, self-contained, easy to use, all-inclusive solution, as long as your backends support the standards [0][1].

> with a completely standardized manifest format for packages

That's wheel [2]. Everything else is deprecated now, except for sdist (which doesn't really matter as it's limited to Python-only packages, and transparently supported by all tools).

> And, for all that is holy, doesn't mess with the global environment so that we have to lie to it with virtual environments.

This would be nice, but it is completely out of hands of the language developers. Different Linux (and other OSs, like OSX) distributions have made different decisions about the python version(s) they bundle by default, and in many cases the bundled versions are critical for some OS-level tools to function properly. Pretty much all interpreted languages either a) have their versions of virtual environments, or b) rely on a single system-wide interpreter.

That being said, there is no need to use virtualenvs if a) you know that you need only one version of Python, and b) you have full control over your system. The best example is service deployments, where you can install a version of your choosing (or start with an appropriate base image), and install everything else on top of it.

> That toolset would have to get overwhelming adoption by the community, to where it retroactively becomes The Python Package Manager (outside of legacy codebases, which would continue to languish). This would probably require endorsement by the Python developers

Why exactly? There are two ways one can work with packages:

1. The majority of users want to be able to install packages.

This is already the case. Pip is overwhelmingly adopted by everyone, with two notable exceptions:

    * Conda, which is mainly used in some niche sub-communities, and which is fully compatible with pip anyway
    * Build tools being used for installing packages, either during development or for deployment in very controlled environments

But both of these exceptions are very specific and narrow in scope.

> Otherwise we'll continue to have: https://xkcd.com/927/

We already don't have that situation. I agree that we did, but the standards are now very clear, and the tools are slowly moving in the right direction.

[0] https://peps.python.org/pep-0518/

[1] https://peps.python.org/pep-0660/

[2] https://realpython.com/python-wheels/

I find python packages quite easy to work with :) plus they're similar to ruby gems.

You create a venv or bundle, list requirements in a text file, then ask it to install things for you.

And if you need custom stuff, you can just pip install a .whl file, too.

I have yet to encounter a case where it's not working as expected, so my answer would be that python isn't getting fixed because it's not broken.

wontfix, works for me

KronisLV · 4 years ago

> You create a venv or bundle, list requirements in a text file, then ask it to install things for you.

Just last week I had problems with a project that I hadn't touched in a while.

After installing Python 3 on a new computer (and making sure that pip is installed) I found that my scripts broke because "pip install" was no longer a thing. I now needed to do something like "python -m pip install".

Not a big issue, just a reminder that things are still improving for the better.

That said, whenever there's native code involved, things can get tricky (especially in Alpine based containers with musl instead of glibc).

That does apply to pretty much everything, just yesterday I also discovered that Ruby was slowing down builds 2-3x because of sassc install being really slow after an update. Then again, the whole library it depends on is deprecated so there's additional migration pain to be had soon.

And don't even get me started on database drivers and their prerequisites in each environment!

That said, even if something like Maven seems (sometimes) better due to no native extensions, I'm still happy that Python/Ruby/Node package managers work as well as they do. Sure beats copying random code manually.

rcarmo · 4 years ago

Your PATH is broken and you’re not finding the pip wrapper for your install. If you need to install Python manually, use pyenv, which actually makes sure pip gets into your PATH to avoid that kind of thing.

samwillis · 4 years ago

Mostly agree, however the only area I think the “official” tools could improve upon is split dependencies between requirements.txt (top level requirements), dev-requirements.txt (top level dev only requirements), and requirements-freeze.txt (all requirements versions frozen for deployment). Currently you have to be careful to ensure your requirements-freeze.txt doesn’t end up with dev dependency’s in it.

fxtentacle · 4 years ago

In the industries where I worked as a C++ coder, it was quite common to leave debugging symbols in the binaries so that customers could have better stack traces for their error diagnostics. On OS X, it is still quite common to ship production apps in a way that allows you to "symbolicate" stack dumps after the fact.

What I'm trying to say: In most production uses, the line between dev and release dependencies is so blurred that it almost doesn't exist.

PaulHoule · 4 years ago

It depends how complex your build is.

Instead of solving dependencies, pip just starts installing stuff and it tries to backtrack if it paints itself into a corner but it frequently gets stuck.

If you dependencies are wheels it is not so bad, in fact with the right software you can download the dependency list of a wheel without downloading a wheel do you could do what real dependency management software (maven) does and download and resolve the dependency graph before installing anything.

With eggs you are SOL because you have to run a Python script to determine the dependencies and if you run enough of those one will have a side effect or two.

thiht · 4 years ago

> You create a venv or bundle

Here’s your dumpster fire. I can’t figure out why this crappy venv thing has to exist in Python when it doesn’t exist anywhere else.

agumonkey · 4 years ago

It's really not special. Having a dedicated ~isolated space for a dependency graph is normal. Better than overwriting global libs.

rcarmo · 4 years ago

Actually, it does. You can do the same in Java, Go, and plenty of other languages. Pretty much every language I’ve used has a way to have multiple runtimes/environments coexist on the same machine, referred to by PATH tweaks or specific environment variables.

zimpenfish · 4 years ago

> this crappy venv thing has to exist in Python when it doesn’t exist anywhere else.

Well, there's RVM[1] for Ruby, maven-env[2] for maven, and perl5-virtualenv[3] for Perl.

[1] https://rvm.io [2] https://github.com/AlejandroRivera/maven-env [3] https://github.com/orkunkaraduman/perl5-virtualenv

fxtentacle · 4 years ago

Apparently, you're new to back-end programming? We have similar "locally installed package" abstractions in every major language there:

C++ -> vcpkg

Ruby -> bundler

Node -> npm vendor

Go -> Go Modules

Java -> Maven

If you disagree, please explain how Ruby's bundler caching dependencies in a vendor subfolder differs from Python's venv caching dependencies in a subfolder and why one of them is a dumpster fire that and the other one doesn't exist.

ochronus · 4 years ago

What's crappy about virtualenvs?

stackbutterflow · 4 years ago

This thread is textbook trolling to get an answer:

```

- I discovered that you'd never get an answer to a problem from Linux Gurus by asking. You have to troll in order for someone to help you with a Linux problem.

- For example, I didn't know how to find files by contents and the man pages were way too confusing. What did I do? I knew from experience that if I just asked, I'd be told to read the man pages even though it was too hard for me.

- Instead, I did what works. Trolling. By stating that Linux sucked because it was so hard to find a file compared to Windows, I got every self-described Linux Guru around the world coming to my aid. They gave me examples after examples of different ways to do it. All this in order to prove to everyone that Linux was better.

- ion has quit IRC (Ping timeout)

- brings a tear to my eye... :') so true..

- So if you're starting out Linux, I advise you to use the same method as I did to get help. Start the sentence with "Linux is gay because it can't do XXX like Windows can". You will have PhDs running to tell you how to solve your problems.

Now it's only a matter of time until someone automates that using GPT3.

I mean we already have AI-automated "email trolling" where they use GPT3 to produce an introduction sentence which suggests that the sender took at least the time to look at my homepage before bothering me, but in reality it's all automated and they are just sending more personal-looking spam than the next guy.

blooalien · 4 years ago

I find it highly offensive on multiple levels that A) people think that sort of behavior is necessary or acceptable, and B) that there are people out there who reinforce that belief by being the ass-hat who answers valid questions with "RTFM!". If someone has trouble understanding man pages, that's reasonable. Turn them on to other (more easily parsed by "average folk") sources of the same information, and request that they "pay it forward" in similar fashion.

brundolf · 4 years ago

mr_mitm · 4 years ago

What do you mean? You type `pip install west` and a binary will appear in `~/.local/bin/west`. This directory needs to be part of your $PATH variable. Done.

Using virtual envs or pipx or poetry or whatever are non-standard.

Packaging Python projects, however, ... don't get me started on the whole `setup.py`, `setup.cfg`, `pyproject.toml` debacle. This article has more information about it, but the fact that this is supposed to be the short version makes it even more infuriating: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated...

kevin_thibedeau · 4 years ago

What's really frustrating is how little information is out there for setting up a pyproject.toml. The community has really dropped the ball on their new shiny.

You are right to an extent, but this is because this "shiny" has been transforming since it was introduced.

Originally, it was meant as a central point to define your build system requirements [0]; for this purpose it needs to include only two lines:

    [build-system]
    # Minimum requirements for the build system to execute.
    requires = ["setuptools", "wheel"]  # PEP 508 specifications.

Then, package managers like pipenv and poetry started using it as a central place to storing other project metadata like dependencies, description etc. Most package managers now have their own versions of that functionality, and it is currently being standardised to a common form [1].

Finally, many other projects have started adding support for keeping its configurations in pyproject.toml. Some (like black) don't even support any other form, while others (like flake8, and until recently mypy) are resisting this trend; but it is already so prevalent that it can be considered the standard.

[1] https://peps.python.org/pep-0621/

julienpalard · 4 years ago

I don't see the dumpster fire here:

$ python -m pip install west

That's it, no venv, no sudo.

pip will install it in ~/.local/ so you'll need a:

    PATH="$PATH:$HOME/.local/bin"

to get it to your path.

nicoco · 4 years ago

On my debian stable, if I want this behaviour I actually have to add `--user`.

But that's not the recommended way to install pypi packages you want to use, with debian at least. This may install dependencies that will break (for your user) some debian packages depending on other versions of said dependencies. The easy way to go is `pipx install west`.

EDIT: --user is not necessary anymore. I think this does not alter the validity of the rest of my comment.

Actually, you shouldn’t need to add that flag, AFAIK it’s been the default for a while for non-root users. But both Debian and Ubuntu tend to package their Pythons to rely on .deb packages and have been somewhat inconsistent over the years.

Does it try to install packages in your system paths and fails due to lack of permissions?

   pip —user

is suicide. You can be doing everything right, using venv’s, conda and all that, install one package in a user directory and blam! You just broke every Python you use because that package is installed in all your environments.

It’s a bad idea right up there with Linux distros having python2, python3, python3.1, puthon3.2, python3.3, …. They give up the ease of use of a command line application where you can tell people what to type for the mind numbing complexity of a GUI application where it takes a 1500 page book to explain how to do anything in Microsoft Word because now you can’t tell people what to type to run and install things.

randyrand · 4 years ago

why use -m?

viraptor · 4 years ago

It ensures you're installing into environment using your current python. Useful in case the op already has a messed up environment where default pip doesn't use default python.

It's a good habit which ensures pip runs with the same interpreter as your `python` command.

One example (but there's many): On Windows a `pip install --upgrade pip` can't work as the OS would lock the `pip` executable, while a `python -m pip install --upgrade pip` works as the OS would lock python instead.

I also encontered environments (CI runners, things like this) with `python` and pip installed, installed but no `pip` shortcut.

More: https://snarky.ca/why-you-should-use-python-m-pip/

shantnutiwari · 4 years ago

Why? Just read the comments here. A combination of hubris and misunderstanding of how hard it is .

"Just do pip install" --> Whenever I hear this comment, I know Im talking to someone who has never used scientific libraries like numpy, scipy etc. Never seen the problem of dependency versions going into a mess because Pip by default doesnt pin dependencies (Poetry does, but it is not standard).

Python packaging is a mess because for some weird reason that baffles me, a large majority won't even admit tehre is a problem. And they will start jumping on you and calling you an idiot if you say there is. A lot of gaslighting going on here.

simonw · 4 years ago

I think Python package management is pretty good now. The absolute toughest bit is learning how to work with virtual environments, but once you have a string understanding of that I think the system works very well.

__d · 4 years ago

Using virtualenv? Or venv? Or poetry? Or pyenv? Or pipenv? Or conda? Or something-else-I-can't-think-of-right-now?

It's still a mess. Arguably, a worse mess than it used to be.

The first two are the same, the rest are effectively different abstractions over them. (apart from pyenv which is not for packages) The whole thing about choices in python packaging became more of a meme than reality. There are specific issues (pip itself not having a lock file equivalent), but they're often blown out of proportions.

I had that convo few days ago. What surprises me is that pip venv were almost ok. But since there's twenty thousands new ways that popped all at once. And I can't understand which does what better.

bb88 · 4 years ago

poetry and pipenv both use virtualenvs under the hood, wrapping it.

They provide dep management. Pip/virtualenv never did dep management really. There was requirements.txt sure, but you had to manage that yourself.

smnrchrds · 4 years ago

I use poetry and forget the rest of them exist. It has worked well so far.

They all change PATH somehow. Which one you use depends on your use case. I, for instance, prefer pyenv for having a runtime and package set that covers most of my projects and virtualenvs for test sandboxes of new package versions.

Conda, for instance, is great for ML and scientific given the extra niceties for replicating environments, but most specifically because you get package sets optimized for your analysis needs.

franga2000 · 4 years ago

I really don't see the problem here - you have two options, both of which involve one install command and at most a one-step setup process:

1. A user install: `pip install [package]` and make sure "$HOME/.local/bin" is on your PATH

2. A global install: `sudo pip install [package]` - it will be installed to a dir on your path already (/usr/bin I think)

As for why pip si not ideal for installing software: it's not supposed to be. It's a Python package manager, not a software package manager. It's meant to install libraries for the Python interpreter to import, not commands for you to run. Of course, people do often library managers to install software (npm, cargo, go...), but the experience is the same in all of them - either you install with sudo and it "just works", but might cause problems later, or you install in "local" mode which requires you to add a the package managers's directory to your path.