I found onlyoffice a good replacement. It is certainly better looking.

Every time I see these things, they look cool as hell, I get excited, then I try to get them working on my gaming PC (that has the GPU), I spend 1-2h fighting with python and give up.

Today's issue is that my python version is 3.12 instead of <3.12,>=3.9. Installing python 3.11 from the official website does nothing, I give up. It's a shame that the amazing work done by people like the OP gets underused because of this mess outside of their control.

"Just use docker". Have you tried using docker on windows? There's a reason I never do dev work on windows.

I spent most of my career in the JVM and Node, and despite the issues, never had to deal with this level of lack of compatibility.

More RAM, so less movement of the weights around to generate a token. Most of the speed limit on a LLM is bandwidth of getting the weights around. To a great extent, your token speed is approximately your (model size)/(effective bandwidth). If you need to shuffle the weights into VRAM from main RAM, you halve your speed (bandwidth used both to move into VRAM and out). If you need to pull the weights from disk, even worse.

>AMD also claims its Strix Halo APUs can deliver 2.2x more tokens per second than the RTX 4090 when running the Llama 70B LLM (Large Language Model) at 1/6th the TDP (75W).

That if true is wild.