Readit Newsanst commented on Llamafile lets you distribute and run LLMs with a single file github.com/Mozilla-Ocho/l... · Posted by u/tfinch
You need to upgrade to zsh 5.9+ or run `sh -c ./llamafile`. See the Gotchas section of the README.
Many thanks! Incredibly versatile implementation.
anst commented on Llamafile lets you distribute and run LLMs with a single file github.com/Mozilla-Ocho/l... · Posted by u/tfinch
You need at minimum a stock operating system install of:
- Linux 2.6.18+ (arm64 or amd64) i.e. any distro RHEL5 or newer
- MacOS 15.6+ (arm64 or amd64, gpu only supported on arm64)
- Windows 8+ (amd64)
- FreeBSD 13+ (amd64, gpu should work in theory)
- NetBSD 9.2+ (amd64, gpu should work in theory)
- OpenBSD 7+ (amd64, no gpu support)
- AMD64 microprocessors must have SSSE3. Otherwise llamafile will print an error and refuse to run. This means, if you have an Intel CPU, it needs to be Intel Core or newer (circa 2006+), and if you have an AMD CPU, then it needs to be Bulldozer or newer (circa 2011+). If you have a newer CPU with AVX or better yet AVX2, then llamafile will utilize your chipset features to go faster. No support for AVX512+ runtime dispatching yet.
- ARM64 microprocessors must have ARMv8a+. This means everything from Apple Silicon to 64-bit Raspberry Pis will work, provided your weights fit into memory.
I've also tested GPU works on Google Cloud Platform and Nvidia Jetson, which has a somewhat different environment. Apple Metal is obviously supported too, and is basically a sure thing so long as xcode is installed.
Time to go amd, poor old me, Intel MB Air 2018 (zsh: exec format error, Darwin Kernel Version 22.2.0, MacOS Ventura 13.1).
Reminds me of a fantastic (in many senses) book of Stanislaw Lem... https://www.goodreads.com/book/show/88313.Peace_on_Earth
Definitely. The Invincible would fit as well
https://en.wikipedia.org/wiki/The_Invincible
Big time for Lem.
anst commented on Taichi lang: High-performance parallel programming in Python taichi-lang.org/... · Posted by u/whereistimbo
anst commented on Taichi lang: High-performance parallel programming in Python taichi-lang.org/... · Posted by u/whereistimbo
Yeah. It's a chicken and egg thing. Imagine if resources applied to PyTorch was spent implementing in Julia.
But then there's not enough users... so the cycle continues until one day Julia hits critical mass and a tipping point is reached.
In a winner-takes-all finite world a tipping point may never happen...
Deleted Comment