I've been writing Python forEVER and the one thing I fail to comprehend about it is how the dependency situation on Python only gets worse. It has somehow surpassed pre-nvm Node for being a total dependency clusterfuck.
What are we using now? Pip? Pyenv? Pipenv? Poetry? Hatch? Conda?
Once that's settled, there's invariable some library that has some native buildchain tooling requirement that isn't satisfied and the build process has shat all over my terminal. So now I have to hunt down whatever libraries gcc is complaining about.
Christ, it should be against the law to release a programming language that doesn't have dependency management built-in.
Yes, for local environment management and library dependency configuration.
> Conda?
Not really, unless you're doing data science and are heavily invested in the rest of Anaconda ecosystem.
I don't really understand where the confusion is coming from - Python is used in a wide number of use cases and scenarios, and it is no surprise that there are many different tools in the box, each solving a different subset of problems.
The packaging issue has been all but resolved in recent years, with the advent of wheels, pyproject.toml and a number of other standards.
> So now I have to hunt down whatever libraries gcc is complaining about.
100% I switch between Go and Python a lot and even with Go mods detractions it’s still so much simpler to have everyone using one package manager built into the standard toolchain
This basically means: Use the version of python between 3.8 and 3.9 and use any version higher than 1.4.3 for pandas.
What I like about poetry is that it makes sure that the whole dependency graph of the packages that you add is correct. If it can not solve the graph, then it fails fast and it fails hard, which is a good thing.
This is probably a very bad explainer of what poetry does, but be sure to check it out! :)
Poetry has a major issue with its lockfiles when working with active projects. It generates a top level dependency listing checksum, which causes any two PRs/branches that independently update the top level requirements to conflict with each other.
The other issue with Poetry is that it uses its own pyproject.toml dependency listing format instead of the one standardized in the relevant PEP (https://peps.python.org/pep-0621/). This is understandable for historical reasons (Poetry was first written before this was standardized), but Poetry should have been updated to support the standard format.
A relatively minor issue, but the poetry shell command is also a footgun. It's presented as a way to configure your shell to activate the virtualenv for the project. In reality it's a very slow, barely functional terminal emulator running on top of your terminal, which will cause problems for any programs that assume a working terminal or talk to the tty directly.
poetry shell is also not a term emulator, it's just a subshell with some environment variables setup for your project. Once you are in, it's just a regular shell. If anything is slow, it's where you add or remove a dependency, but it's probably faster than you editing requirements.txt, clearing out your virtualenv and then reinstalling everything again.
Poetry is good, I'll hit a bug in poetry from time to time but it's improving quickly.
Direnv is better, direnv layout will automatically create a hidden virtualenv directory on your work tree and sets up your path when you cd into it. The only downside is it doesn't seem to work on Windows.
I don't see the link between poetry and direnv? Poetry is about solving python's dependency issues, direnv doesn't seem to have anything to do with that
Maybe it is the same, but the idea is to take that selected version of the dependency and store a hash checksum for it, so that one can later get the exact same dependencies. Poetry only does not source setup.cfg, but its tool-specific config file. This way it stays out of the way of any other tool.
In comparison to poetry I think it includes more advanced multi-environment and multi-python-version support and a tox-like testing matrix. It probably gets a little too complex there.
It also works with pyproject.toml
If anyone else has experience with Hatch vs Poetry please share!
I've been using PDM recently and although there have been a few issues I really like just cd'ing into a directory, running "python" and the correct set of packages being available.
I seem to remember that some part of poetry's slowness is due to how the python index works and is therefore a problem shared by all such tools.
That said, I've used both pipenv and poetry, and I had projects where pipenv would simply time out when trying to resolve packages. I haven't seen the same behaviour with poetry (indeed, that was the reason I migrated one project from pipenv to poetry after I just had to give up with the former).
This is interesting; can you expand or point to some documentation? I don’t have this design requirement right not, so I’m trying to understand any growing pains I might be locking myself into
You still need something else to manage your python versions and virtualenvs tho, and then as soon as you've pick a solution for the latter two problems, chances are you'll discover these tools also have a lock files that'll solve the problem of freezing packages and enabling reproducable builds for you.
We are migrating our company codebase to bazel which uses this and it is quite helpful. Hermetic builds are nice and all devs share same versions of packages so reproducing bugs is easy
Shameless plug: I built pip-chill to help me generate less stringent (and shorter) requirements.txt files. Its main use is to remove "noise" - packages required by other packages that you don't care about - but it can also remove version numbers (which is useful for testing against the latest and greatest).
pip-tools is alright but it doesn't support cross-platform (or python version) lockfiles.
poetry is alright but it doesn't support the latest PEP standards, and its slow.
PDM is where it's at; it's fast, has a really responsive maintainer, supports all the latest PEP standards, and has really good cross-platform support.
Readit News
What are we using now? Pip? Pyenv? Pipenv? Poetry? Hatch? Conda?
Once that's settled, there's invariable some library that has some native buildchain tooling requirement that isn't satisfied and the build process has shat all over my terminal. So now I have to hunt down whatever libraries gcc is complaining about.
Christ, it should be against the law to release a programming language that doesn't have dependency management built-in.
Of course, for standard package installation.
> Pyenv?
Yes, if you need multiple Python versions available on the same system.
> Pipenv?
God no. https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
> Poetry? Hatch?
Yes, for local environment management and library dependency configuration.
> Conda?
Not really, unless you're doing data science and are heavily invested in the rest of Anaconda ecosystem.
I don't really understand where the confusion is coming from - Python is used in a wide number of use cases and scenarios, and it is no surprise that there are many different tools in the box, each solving a different subset of problems.
The packaging issue has been all but resolved in recent years, with the advent of wheels, pyproject.toml and a number of other standards.
> So now I have to hunt down whatever libraries gcc is complaining about.
Which has nothing to do with Python.
It is a great tool that I use in all my projects. It effectively solves dependency management for me in a very easy to use way.
Dependency management using requirements.txt used to give me such a headache, now I just have a pyproject.toml that I know works.
This basically means: Use the version of python between 3.8 and 3.9 and use any version higher than 1.4.3 for pandas.What I like about poetry is that it makes sure that the whole dependency graph of the packages that you add is correct. If it can not solve the graph, then it fails fast and it fails hard, which is a good thing.
This is probably a very bad explainer of what poetry does, but be sure to check it out! :)
[1] https://python-poetry.org/
The related issue, https://github.com/python-poetry/poetry/issues/496, has been open for 4 years with no movement.
The other issue with Poetry is that it uses its own pyproject.toml dependency listing format instead of the one standardized in the relevant PEP (https://peps.python.org/pep-0621/). This is understandable for historical reasons (Poetry was first written before this was standardized), but Poetry should have been updated to support the standard format.
A relatively minor issue, but the poetry shell command is also a footgun. It's presented as a way to configure your shell to activate the virtualenv for the project. In reality it's a very slow, barely functional terminal emulator running on top of your terminal, which will cause problems for any programs that assume a working terminal or talk to the tty directly.
poetry shell is also not a term emulator, it's just a subshell with some environment variables setup for your project. Once you are in, it's just a regular shell. If anything is slow, it's where you add or remove a dependency, but it's probably faster than you editing requirements.txt, clearing out your virtualenv and then reinstalling everything again.
Direnv is better, direnv layout will automatically create a hidden virtualenv directory on your work tree and sets up your path when you cd into it. The only downside is it doesn't seem to work on Windows.
You can also constrain the Python version in there if you want.
[1] https://setuptools.pypa.io/en/latest/userguide/quickstart.ht...
https://hatch.pypa.io/latest/
In comparison to poetry I think it includes more advanced multi-environment and multi-python-version support and a tox-like testing matrix. It probably gets a little too complex there.
It also works with pyproject.toml
If anyone else has experience with Hatch vs Poetry please share!
https://hatch.pypa.io/latest/meta/faq/#libraries-vs-applicat...
As does pip these days.
That said, I've used both pipenv and poetry, and I had projects where pipenv would simply time out when trying to resolve packages. I haven't seen the same behaviour with poetry (indeed, that was the reason I migrated one project from pipenv to poetry after I just had to give up with the former).
I might be splitting hairs here, but this seems like an oxymoron: if it's agnostic on anything, it's not really a lock file.
https://github.com/jazzband/pip-tools/issues/1167
I personally find it extremely useful when upgrading dependencies.
Deleted Comment
poetry is alright but it doesn't support the latest PEP standards, and its slow.
PDM is where it's at; it's fast, has a really responsive maintainer, supports all the latest PEP standards, and has really good cross-platform support.
pdm.fming.dev/
https://twitter.com/mkennedy/status/1375242144135270403?lang...