> But surely quite a lot of the 671 GB or whatever is knowledge that is usually irrelevant?

Small correction - It's 671B Parameters - not 671 Gigabytes (doing some rudimentary math if you want to run the entire model in memory it would take ~750GB (671b * fp8 == 8 bits * 1.2 (20% overhead)) = 749.901 GiB)

It's a MoE model so you don't actually need to load all 750gb at once.

I think maybe what you are asking is "Why do more params make a better model?"

Generally speaking its because if you have more units of representation (params) you can encode more information about the relationships in the data used to train the model.

Think of it like building a LEGO city.

A model with fewer parameters is like having a small LEGO set with fewer blocks. You can still build something cool, like a little house or a car, but you're limited in how detailed or complex it can be.

A model with more parameters is like having a giant LEGO set with thousands of pieces in all shapes and colours. Now, you can build an entire city with skyscrapers, parks, and detailed streets.

---

In terms of "is a lot of of irrelevant?" - This is a hot area of research!

It's very difficult currently to know what parameters are relevant and what aren't - there is an area of research called mechanistic interpretability that aims to illuminate this - if you are interested - Anthropic released a good paper called "Golden Gate Claude" on this.

I’ve been running quarto [0] for a few months now and I’m happy with it. Posts are saved as .qmd, with a little bit of special front matter for formatting and tagging. `quarto render` converts the .qmd(s) according to a simple config file. [0] https://quarto.org/

I got into the habit of running every query I do through all of the SOTA models so I can directly compare the results.

This was originally GPT-4, Claude 3 Opus, and Gemini Advanced. I recently added Meta AI when they launched.

Right now I've sent 486 queries through the first three systems.

The clearest pattern to emerge is that Gemini is terrible, not on par with the other two. There hasn't been a single query that it was the only model who did well. Around 1/4 of the time it gives a clearly inferior answer to the others.

But between GPT-4 and Claude it's less clear. 31 of the 486 queries Claude provided a significantly better answer than the other two but 20 times GPT-4 provided the significantly better answer.

I do think that Claude is a slightly better model but right now it's not a clear enough advantage that I'd recommend it generally. I will say you can probably cancel you Gemini subscription if you're using it though.

You can install Midnight Commander on a Mac as well.