Yikes what's the bar for dead simple these days? Even my totally non-technical gamer friends are messing around with ollama because I just have to give them one command to get any of the popular LLMs up and running.
Now of course "non technical" here is still a pc gamer that's had to fix drivers once or twice and messaged me to ask "hey how do i into LLM, Mr. AI knower", but I don't think twice these days about showing any pc owner how to use ollama because I know I probably won't be on the hook for much technical support. My sysadmin friends are easily writing clever scripts against ollama's JSON output to do log analysis and other stuff.
By "too hard" I do not mean getting started with them to run inference on a prompt. Ollama especially makes that quite easy. But as an application developer, I feel these platforms are too hard to build around. The main issues being: getting the correct small enough task specific model and how long it takes to download these models for the end user.
I guess it depends on expectations, if your expectation is an CRUD app that opens in 5 seconds, then sure, it's definitely tedious. People do install things though, the companion app for DJI action cameras is 700mb (which is an abomination, but still). Modern games are > 100gb on the high side, downloading 8-16gb of tensors one time is nbd. You mentioned that there are 663 different models of dsr1-7b on huggingface, sure, but if you want that model on ollama it's just `ollama run deepseek-r1`
As a developer the amount of effort I'm likely to spend on the infra side of getting the model onto the user's computer and getting it running is now FAR FAR below the amount of time I'll spend developing the app itself or getting together a dataset to tune the model I want etc. Inference is solved enough. "getting the correct small enough model" is something that I would spend the day or two thinking about/testing when building something regardless. It's not hard to check how much VRAM someone has and get the right model, the decision tree for that will have like 4 branches. It's just so little effort compared to everything else you're going to have to do to deliver something of value to someone. Especially in the set of users that have a good reason to run locally.
Mac's unified memory makes them (price-) compelling over x86 with GPU(s) for large models, say something over 24-32 GB. But a 32GB Mac doesn't take advantage of that architecture.
(IIRC by default you can use 66% of RAM in < 32GB Metal boxes, and something higher in > 32GB -- though you can override that value via sysctl.)
Macs can also run mlx in addition to gguf, which on these smaller models would be faster. No mention of mlx, or indeed gguf.
The only model tested seems to be a distil of Deepseek R1 with Qwen - which I'd have classified as 'good, but not great'.
Author bemoans all quants of that are > 5GB, which isn't true. Though with 20GB of effective VRAM to play with here, you wouldn't want to be using the Q4 (at 4.6GB).
Author seems to conflate the one-off download cost (time) from hf with on-going performance cost of using the tool.
No actual client-side tooling in play, either, by the looks of it, which seems odd given the claim that local inference is 'not ready as a developer platform'.
The usual starting point for most devs using local LLMs is vscode + continue.dev , where the 'developer experience' is a bit more interesting than just copy-pasting to a terminal.
Criterion (singular) for LLM model expertise appears to be 'text to SQL', which is fair enough if you were writing about applicability of "Local LLM Inference For Text to SQL". I'd have expected the more coding-specific (qwen2.5 coder 14B, codestral, gemma?) models would be more interesting than just one 6GB distil of R1 & Qwen.
Huggingface has some functional search, though https://llm.extractum.io/list/ is a bit better in my experience, as you can tune & sort by size, vintage, licence, max context length, popularity, etc.
I concur that freely available can-run-in-16GB-of-RAM models are not as good as Claude, but disagree that the user experience is as bad as painted here.
Download the model in background. Serve the client with an LLM vendor API just for the first requests, or even using that same local LLM installed on your own servers (likely cheaper). By doing so, in the long run the inference cost is near-zero and allows to use LLMs in otherwise impossible business models (like freemium).
Personally, I only use locally run models when I absolutely can’t have the prompt/context uploaded to a cloud. For anything else, I just use one of the commercial cloud hosted models. The ones I‘m using are way faster and better in _every_ way except privacy. Eg if you are ok to spend more, you can get blazing fast DeepSeek v3 or R1 via OpenRouter. Or, rather cheap Claude Sonnet via Copilot (pre-release also has Gemini 2.5 Pro btw).
I’ve gotten carried away - I meant to express that using cloud as a fallback for local models is something I absolutely don’t want or need, because privacy is the whole and only point to local models.
There's two general categories of local inference:
- You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server.
- You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running).
I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough with the ecosystem to handle some of the issues he ran into when attempting the second (e.g., exactly which model to use). But the second has a bunch of unique constraints--you want things to just work for your users, after all.
I've done in-browser neural network stuff in the past (back when using TensorFlow.js was a reasonable default choice) and based on the way LLM trends are going I'd guess that edge device LLM will be relatively reasonable soon; I'm not quite sure that I'd deploy it in production this month but ask me again in a few.
Relatively tightly constrained applications are going to benefit more than general-purpose chatbots; pick a small model that's relatively good at your task and train it on enough of your data and you can get a 1B or 3B model that has acceptable performance, let alone the 7B ones being discussed here. It absolutely won't replace ChatGPT (though we're getting closer to replacing ChatGPT 3.5 with small models). But if you've got a specific use case that will hold still enough to deploy a model it can definitely give you the edge versus relying on the APIs.
I expect games to be one of the first to try this: per-player-action API costs murder per-user revenue, most of the gaming devices have some form of GPU already, and most games are shipped as apps so bundling a few more GB in there is, if not reasonable, at least not unprecedented.
Very interesting. I had not thought about gaming at all but that makes a lot of sense.
I also agree the goal should not be to replace ChatGPT. I think ChatGPT is way overkill for a lot of the workloads it is handling. A good solution should probably use the cloud LLM outputs to train a smaller model to deploy in the background.
Inference using an API costs money. Not a lot of money, per million tokens, but it adds up if you have a lot of tokens...and some of the obvious game uses really chew through the tokens. Like chatting with a character, or having the NPC character make decisions via reasoning model. Can easily make the tokens add up.
Games, on the other hand, are mostly funded via up-front purchase (so you get the money once and then have to keep the servers running) or free to play, which very carefully tracks user acquisition costs versus revenue. Most F2P games make a tiny amount per player; they make up the difference via volume (and whales). So even a handful of queries per day per player can bankrupt you if you have a million players and no way to recoup the inference cost.
Now, you can obviously add a subscription or ongoing charge to offset it, but that's not how the industry is mostly set up at the moment. I expect that the funding model will change, but meanwhile having a model on the edge device is the only currently realistic way to afford adding an LLM to a big single player RPG, for example.
What if I charge "whales" in games to talk to an anime girl? Maybe I'll only let you talk to her once a day unless you pay me like a kissing booth for every convo. There's going to be some predatory stuff out there, I can see what the GP is talking about with games.
For a while basically any mobile or browser freemium game you tried would have progress timers for building things or upgrading things and they'd charge you Actual Money to skip the wait. That's kind of out of fashion now though some games still do it.
The only bar to using local is having the hardware and downloading the model. I find it nominally easier to use than using the openAI API since the local API isn’t picky about some of the fields (by default). Agentic flows can use local 90percent of the time and reach out to god when they need divine insight, saving 90 percent of token budgets and somewhat reducing external exposure, though I prefer to keep everything locally if possible. It’s not hard to run a 70b model locally, but the queue can get backed up with multiple users unless you have very strong hardware. Still, you can shift overflow to the cloud if you want.
Using a 7B model on a M2 Max also isn’t quite the most impressive way to locally run an LLM. Why not use QwQ-32 and let it give some commercial non-reasoning models a run for their money?
Exactly. You want to come close to maxing out your RAM for model+context. I've run Gemma on a 64GB M1 and it was pretty okay, although that was before the Quantization-Aware Training version released last week, so it might be even better now.
Basically you let a very small model speculate on the next few tokens, and the large model then blesses/rejects those predictions. Depending on how well the small model performs, you get massive speedups that way.
The small model has to be as close to the big model as possible - I tried this with models from different vendors and it slowed generation down by x3 or so. So, you need to use a small Qwen 2.5 with a big Qwen 2.5, etc
How exactly does this give a speedup? If you have to wait for the large model to confirm the small model's predictions, wouldn't it always be slower than just running the large model?
Local inferencing is synonymous with privacy for me. There is no universe until laws get put into effect where your LLM usage online is private as it stands now. I suspect most of these companies are going to be putting in a Microsoft Clippy style assistant in soon that will act as a recommendation/ad engine very soon, and this of course requires parsing every convo you've ever had. Paid tier may remove Clippy, but boy oh boy the free tier (which most people will use) won't.
Clippy is coming back guys, and we have to be ready for it.
I‘ve configured Little Snitch to only allow it access to huggingface. I think for updates I need to reset LS to „ask for each connection“ or sthg like that.
If you want privacy, use a local models and an open-source chat interface such as OpenWeb-UI or Jan. (avoid proprietary systems such as Msty or LM Studio).
Can anyone vouch for this? I (personally) don't mind that it's closed source, but I've never heard of it and can't find much about it. Website makes it look fantastic though, so I'm intrigued. But am hesitating at giving it all of my API keys..
Llamafile is great and love it. I run all my models using it and it’s super portable, I have tested it on windows and linux, on a powerful PC and SBC. It worked great without too my issues.
It takes about a month for the features from llama.cpp to trickle in. Also figuring the best mix of context length size to vram size to desired speed takes a while before it gets intuitive.
Readit News
Now of course "non technical" here is still a pc gamer that's had to fix drivers once or twice and messaged me to ask "hey how do i into LLM, Mr. AI knower", but I don't think twice these days about showing any pc owner how to use ollama because I know I probably won't be on the hook for much technical support. My sysadmin friends are easily writing clever scripts against ollama's JSON output to do log analysis and other stuff.
As a developer the amount of effort I'm likely to spend on the infra side of getting the model onto the user's computer and getting it running is now FAR FAR below the amount of time I'll spend developing the app itself or getting together a dataset to tune the model I want etc. Inference is solved enough. "getting the correct small enough model" is something that I would spend the day or two thinking about/testing when building something regardless. It's not hard to check how much VRAM someone has and get the right model, the decision tree for that will have like 4 branches. It's just so little effort compared to everything else you're going to have to do to deliver something of value to someone. Especially in the set of users that have a good reason to run locally.
Mac's unified memory makes them (price-) compelling over x86 with GPU(s) for large models, say something over 24-32 GB. But a 32GB Mac doesn't take advantage of that architecture.
(IIRC by default you can use 66% of RAM in < 32GB Metal boxes, and something higher in > 32GB -- though you can override that value via sysctl.)
Macs can also run mlx in addition to gguf, which on these smaller models would be faster. No mention of mlx, or indeed gguf.
The only model tested seems to be a distil of Deepseek R1 with Qwen - which I'd have classified as 'good, but not great'.
Author bemoans all quants of that are > 5GB, which isn't true. Though with 20GB of effective VRAM to play with here, you wouldn't want to be using the Q4 (at 4.6GB).
Author seems to conflate the one-off download cost (time) from hf with on-going performance cost of using the tool.
No actual client-side tooling in play, either, by the looks of it, which seems odd given the claim that local inference is 'not ready as a developer platform'.
The usual starting point for most devs using local LLMs is vscode + continue.dev , where the 'developer experience' is a bit more interesting than just copy-pasting to a terminal.
Criterion (singular) for LLM model expertise appears to be 'text to SQL', which is fair enough if you were writing about applicability of "Local LLM Inference For Text to SQL". I'd have expected the more coding-specific (qwen2.5 coder 14B, codestral, gemma?) models would be more interesting than just one 6GB distil of R1 & Qwen.
Huggingface has some functional search, though https://llm.extractum.io/list/ is a bit better in my experience, as you can tune & sort by size, vintage, licence, max context length, popularity, etc.
I concur that freely available can-run-in-16GB-of-RAM models are not as good as Claude, but disagree that the user experience is as bad as painted here.
I’ve gotten carried away - I meant to express that using cloud as a fallback for local models is something I absolutely don’t want or need, because privacy is the whole and only point to local models.
- You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server.
- You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running).
I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough with the ecosystem to handle some of the issues he ran into when attempting the second (e.g., exactly which model to use). But the second has a bunch of unique constraints--you want things to just work for your users, after all.
I've done in-browser neural network stuff in the past (back when using TensorFlow.js was a reasonable default choice) and based on the way LLM trends are going I'd guess that edge device LLM will be relatively reasonable soon; I'm not quite sure that I'd deploy it in production this month but ask me again in a few.
Relatively tightly constrained applications are going to benefit more than general-purpose chatbots; pick a small model that's relatively good at your task and train it on enough of your data and you can get a 1B or 3B model that has acceptable performance, let alone the 7B ones being discussed here. It absolutely won't replace ChatGPT (though we're getting closer to replacing ChatGPT 3.5 with small models). But if you've got a specific use case that will hold still enough to deploy a model it can definitely give you the edge versus relying on the APIs.
I expect games to be one of the first to try this: per-player-action API costs murder per-user revenue, most of the gaming devices have some form of GPU already, and most games are shipped as apps so bundling a few more GB in there is, if not reasonable, at least not unprecedented.
I also agree the goal should not be to replace ChatGPT. I think ChatGPT is way overkill for a lot of the workloads it is handling. A good solution should probably use the cloud LLM outputs to train a smaller model to deploy in the background.
Games, on the other hand, are mostly funded via up-front purchase (so you get the money once and then have to keep the servers running) or free to play, which very carefully tracks user acquisition costs versus revenue. Most F2P games make a tiny amount per player; they make up the difference via volume (and whales). So even a handful of queries per day per player can bankrupt you if you have a million players and no way to recoup the inference cost.
Now, you can obviously add a subscription or ongoing charge to offset it, but that's not how the industry is mostly set up at the moment. I expect that the funding model will change, but meanwhile having a model on the edge device is the only currently realistic way to afford adding an LLM to a big single player RPG, for example.
Basically you let a very small model speculate on the next few tokens, and the large model then blesses/rejects those predictions. Depending on how well the small model performs, you get massive speedups that way.
The small model has to be as close to the big model as possible - I tried this with models from different vendors and it slowed generation down by x3 or so. So, you need to use a small Qwen 2.5 with a big Qwen 2.5, etc
I'm sure someone is watching their internet traffic, but I don't.
I take the risk now, but I ask questions about myself, relationships, conversations, etc... Stuff I don't exactly want Microsoft/ChatGPT to have.
Clippy is coming back guys, and we have to be ready for it.
https://github.com/janhq/jan
https://github.com/open-webui/open-webui
https://msty.app/
Some models have even a 0.5B draft model. The speed increase is incredible.
https://github.com/Mozilla-Ocho/llamafile
It takes about a month for the features from llama.cpp to trickle in. Also figuring the best mix of context length size to vram size to desired speed takes a while before it gets intuitive.