Chrome with 12 tabs open
gonna need quantum computing once you breach in to the mid 20s
Readit NewsDead Comment
Dead Comment
Dead Comment
Deleted Comment
Dead Comment
pajeets commented on Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run? · Posted by u/pajeets
pajeets commented on Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run? · Posted by u/pajeets
Try llama.cpp with the biggest LLM you can find.
need a 3090 at least for that
pajeets commented on Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run? · Posted by u/pajeets
There are a variety of other LLM inference implementations that can run on CPU as well.
[0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su...
[1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...
what model can i run on 1TB and how many tokens per second ?
for instance Nvidia Nemotron Llama 3.1 quantized at what speed ? ill get a GPU too but not sure how much VRAM I need for the best value for your buck