PyPy was first. It was indeed easier in the era when the Cheese Shop was called the Cheese Shop, and not PyPi.
[edit]: After some searching, I notice that PEP 301 is actually dated Oct 2002, so perhaps I'm wrong. The public/advertised name was Cheese Shop for quite some time after that, but apparently the name PyPi was at least suggested much earlier than I thought.
I've used pypy on large codebases for years. Generally it's fine so long as you don't need any of the packages that are thin wrappers around C/fortran. It seems a lot of maintainers these days are pretty good about considering it as a target
On the other hand the memory footprint can be painful - not just the deferred garbage collection of things like weakrefs and closed files, but even regular objects. A while back I had hope that the faster cpython project would somewhat remove our need to use it, just so we could have a lower memory footprint, but that seems to have stalled
Why? What specifically caused you not to seek out alternatives? PyPy and libraries with CFFI seem to work okay together, but it can run python native code nearly as fast as C, so I always look for a pure python solution rather than depend on a C library.
They likely mean some of the numerical packages that call out to BLAS/LAPACK or whatever as all that was originally written in Fortran. I think a lot has been converted to C now.
numpy / scipy both use fortran. Scipy has fortran as a build dependency (ie a large portion of it depends on fortran), and numpy can be built without fortran, but has large portions of optimizations in fortran.
anyone who's ever used numpy/scipy has been using python integrated with fortran.
I just recently made some Python wrappers for a Modern Fortran optimization library for derivative free optimization. https://github.com/libprima/prima
It basically had to go through Fortran -> C -> C++ (with pybind11) -> Python. At one point we had a slightly simpler setup with Fortran -> C -> Python (with ctypes), but ended up going with pybind11 after some discussion with the SciPy folks who steered us away from ctypes.
Not the OP, but I have written a fair few python wrappers around old Fortran code. There is a pretty great tool called f2py which automates almost all of the hard work. If the Fortran code isn't to weird it can completely autogenerate all the code you need to call your fortran function from python.
Fortran is actually pretty okay if you're doing matrix or vector operations! Builtin cross products, you choose how the array is indexed when you create it, can enforce functional purity to help you parallelize, etc
It is hot garbage for almost anything that isn't math though - which is okay, because it's been focused math from the start - strings and parsing an unstructured input is an exercise in pain for example. And the ecosystem is heavily geared toward math and scientific computing, so you will find yourself rolling your own stuff quite often if you deviate from that niche
It is a bit sad that PyPy great achievements are mostly ignored, and it took Microsoft and Facebook stepping in, Julia and Mojo arriving into the scene, AI GPU frameworks JITs, for finally CPython to take having a JIT seriously.
If someone suggested switching to a new C++ compiler with 99% of the features of the C++ standard with the selling point that it ran template metaprograms 100x faster, could you convince your company to switch? In the setting where python shines and performance matters, python is serving as the metalanguage executed at compile time for a strictly typed compiled language such as SQL, torch graph traces, jax, or z3
pypy has never kept up with cPython features and releases. it was always well behind the curve and did not provide essential behaviors in a timely fashion. plus it can't run any C extensions or cython code (or has to run them with emulations that erase the speed benefits), which takes a big chunk out of the available ecosystem. Example: the situation with numpy: https://doc.pypy.org/en/latest/faq.html#what-about-numpy-num...
I had always assumed it was because they had exposed far too many internals over the years, making a JIT that was backwards compatible with the majority of existing code, an extremely difficult undertaking.
I assumed it was because they explicitly did not have performance as a goal, probably because performance is so bad they wanted to have a good answer and "performance is a non-goal" is the best they could do. Psychologically that is much easier to digest than "yes it's awful, sorry".
I don't think backwards compatibility can really be the reason. Python does not care that much about backwards compatibility, as seen from the extensive list of features they remove in every release:
I don't see any reason for Python to want to have JIT.
There are already languages that are more suited for that kind of model, that have a long history of incremental improvement of JIT compiler design and implementation. Why is Python trying to compete in the field where it's almost certainly guaranteed to lose, and is almost certainly guaranteed to win nothing?
Having an interpreter is great for some things. Especially, if you want to have bindings to third-party libraries, which is something Python does a lot.
----
My interpretation of the events is very different from yours. I believe that Python became swarmed by people who really, really want Java, but they feel that Java isn't cool for some superficial reason. And guided by the fear of crowd opinion, they won't just use Java and be content with it. Instead, they slowly make what used to be cool into a very bad clone of Java.
And the reason why the big names you mentioned have anything to do with JIT development is because they want to ride this wave of insecure and self-doubting developers to score popularity points, and, eventually, to control the popular technology. Esp. in the case of Microsoft, it's a multi-prong offensive. They've installed their people in Python Foundation, various SIGs related to Python. They've made CPython builds and CI run on their infrastructure, preventing it from choosing free tools for their development process... Microsoft has a bad rep with people who used to like Python, and that's why they don't rebrand it into MS-Python, but really, if they wanted to, they could certainly do it.
The next versions of CPython are expanding in two very important areas for speed. The free-threading (NoGIL) project will allow true multithreading support and the Faster CPython is trying to speedup Python and are developing a JIT. Exciting times!
for several years I used to use pypy to get the dev speed of python on the kinda crunchy problems that you'd normally need to work in a classic compiled language for.
I remember it fondly. At the time, you could find the devs on freenode and the few times I bumped into something being slower than expected they jumped on my informal bug reports and quickly recreated it and fixed it etc. Awesome! :)
Yeah, it's like they're shooting themselves in both feet by just naming their project badly. What a pity.
It could have been worse. Years ago a promising audio format was competing with mp3. It was better quality at any given compression and open source. The creators named it "Ogg Vorbis". Like some kind of dungeons and dragons character or something. I haven't seen it around for a long time, I think that's because the name was so stupid.
Tangentially related but pipx is a silent hero in the quest for better python environment management. Takes care of the problems with bootstrapping your environment manager of choice without installing it in the python root.
Readit News
Naming things is hard but it would be nice if people tried a bit harder to avoid clashes at least within the same ecosystem.
[edit]: After some searching, I notice that PEP 301 is actually dated Oct 2002, so perhaps I'm wrong. The public/advertised name was Cheese Shop for quite some time after that, but apparently the name PyPi was at least suggested much earlier than I thought.
https://peps.python.org/pep-0301/
On the other hand the memory footprint can be painful - not just the deferred garbage collection of things like weakrefs and closed files, but even regular objects. A while back I had hope that the faster cpython project would somewhat remove our need to use it, just so we could have a lower memory footprint, but that seems to have stalled
This is precisely where my attempt to use Pypy failed.
anyone who's ever used numpy/scipy has been using python integrated with fortran.
It basically had to go through Fortran -> C -> C++ (with pybind11) -> Python. At one point we had a slightly simpler setup with Fortran -> C -> Python (with ctypes), but ended up going with pybind11 after some discussion with the SciPy folks who steered us away from ctypes.
It is hot garbage for almost anything that isn't math though - which is okay, because it's been focused math from the start - strings and parsing an unstructured input is an exercise in pain for example. And the ecosystem is heavily geared toward math and scientific computing, so you will find yourself rolling your own stuff quite often if you deviate from that niche
It always seemed to me like it a "free" way to make it faster. Why would it matter whether it was endorsed by the BDFL or PSF?
I don't think backwards compatibility can really be the reason. Python does not care that much about backwards compatibility, as seen from the extensive list of features they remove in every release:
https://docs.python.org/3/whatsnew/3.12.html#removed
There are already languages that are more suited for that kind of model, that have a long history of incremental improvement of JIT compiler design and implementation. Why is Python trying to compete in the field where it's almost certainly guaranteed to lose, and is almost certainly guaranteed to win nothing?
Having an interpreter is great for some things. Especially, if you want to have bindings to third-party libraries, which is something Python does a lot.
----
My interpretation of the events is very different from yours. I believe that Python became swarmed by people who really, really want Java, but they feel that Java isn't cool for some superficial reason. And guided by the fear of crowd opinion, they won't just use Java and be content with it. Instead, they slowly make what used to be cool into a very bad clone of Java.
And the reason why the big names you mentioned have anything to do with JIT development is because they want to ride this wave of insecure and self-doubting developers to score popularity points, and, eventually, to control the popular technology. Esp. in the case of Microsoft, it's a multi-prong offensive. They've installed their people in Python Foundation, various SIGs related to Python. They've made CPython builds and CI run on their infrastructure, preventing it from choosing free tools for their development process... Microsoft has a bad rep with people who used to like Python, and that's why they don't rebrand it into MS-Python, but really, if they wanted to, they could certainly do it.
No matter how things are supposed to be, that is not how bootcamp, self called "engineers", do their programming.
Also as proven by other ecosystems, with JIT, AOT, REPLs, in the box, having an interpreter like CPython isn't a requirement for anything.
As for your MS jab, it was called IronPython.
I remember it fondly. At the time, you could find the devs on freenode and the few times I bumped into something being slower than expected they jumped on my informal bug reports and quickly recreated it and fixed it etc. Awesome! :)
This was very confusing, especially because they mentionned pipx, which seems related to PyPi, so it was plausible this was about PyPi.
It could have been worse. Years ago a promising audio format was competing with mp3. It was better quality at any given compression and open source. The creators named it "Ogg Vorbis". Like some kind of dungeons and dragons character or something. I haven't seen it around for a long time, I think that's because the name was so stupid.
1. http://opus-codec.org/comparison/
Spotify is using it in their standalone clients.
Deleted Comment
uv + pyproject.toml + pyenv virtualenvs work pretty flawlessly got me.
> an experimental restricted-Python to C++ programming language compiler