Speculative decoding! It makes inference a LOT faster.

Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster.

If it's wrong, you have to generate it again the normal way (a lot slower than just checking it). Usually, it's correct, so inference is a lot faster.

> What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?

It is only useful for inference and doesn't help with pretraining. Which actually points to speculative decoding not being sufficiently general, as the same underlying property (some sequences of tokens are easy to predict) could be exploited for training as well. See here: https://goombalab.github.io/blog/2025/hnet-future/#d-footnot...

It could be a better draft model than separately trained EAGLE etc for speculative decoding.

Background:

LLMs take your input, upscale it into a very high dimensional space, and then downscale it back to 1D at the end. This 1D list is interpreted as a list of probabilities -- one for each word in your vocabulary. i.e f(x) = downscale(upscale(x)). Each of downscale() and upscale() are parameterized (billions of params). I see you have a gamedev background, so as an example: bezier curves are parameterized functions where bezier handles are the parameters. During training, these parameters are continuously adjusted so that the output of the overall function gets closer to the expected result. Neural networks are just really flexible functions for which you can choose parameters to get any expected result, provided you have enough of them (similar to bezier curves in this regard).

---

When training, you make an LLM learn that

I use arch = downscale(upscale(I use))

If you want to predict the next word after that, you do next in sequence the following:

I use arch btw = downscale(upscale(I use arch))

Now, multi-token prediction is having two downscale functions, one for each of the next two words, and learning it that way, basically, you have a second downscale2() that learns how to predict the next-to-next word.

i.e in parallel:

I use arch = downscale1(upscale(I use))

I use ____ btw = downscale2(upscale(I use))

However, this way you'll need twice the number of parameters downscale needs. And if you want to predict more tokens ahead you'll need even more parameters.

What Qwen has done, is instead of downscale1 and downscale2 being completely separately parameterized functions, they set downscale1(.) = lightweight1(downscale_common(.)) and downscale2(.) = lightweight2(downscale_common(.)). This is essentially betting that a lot of the logic is common and the difference between predicting the next and next-to-next token can be captured in one lightweight function each. Lightweight here, means less parameters. The bet paid off.

So overall, you save params.

Concretely,

Before: downscale1.params + downscale2.params

After: downscale_common.params + lightweight1.params + lightweight2.params

Edit: its actually downscale_common(lightweight()) and not the other way around as I have written above. Doesn't change the crux of the answer, but just including this for clarity.

The best primer I've seen is Andrej Karpathy's first video in his "zero to hero" series. It's worth following along with your own practice.

https://karpathy.ai/zero-to-hero.html

Unfortunately, no. The industry is moving super quickly, and spinning up new ideas on the backs of old ones at a fast rate. If you want to understand what's going on, I think the best thing to do is some intro courses, train and design some smaller models directly, get a list of core papers and concepts from Claude/Chat/Gemini, and then as you read something like this, if you don't know the acronym (In this case: MTP = Multi Token Prediction), search it up, and see if you have the basis for understanding what it's about. If not read up on the precursors.

Unlike many disciplines, AI is an arena that doesn't have a lot of intuitive simplified models that are accurate -- most of the simplified models available do not accurately describe what's going on enough to reason about and understand them. So, you just have to start reading!

For me, ChatGPT or any of the other current thinking models are very useful for this type of stuff. I just ask to explain it on my level and then I can ask questions for clarification.

The following was generated by chatG5:

    Qwen3-Next — A family of large language models from Qwen (Alibaba).  
    DeepSeek R1 — Another large open-source language model from DeepSeek AI.  
    Linear attention — A type of transformer attention that scales linearly with sequence length, making long-context processing cheaper.  
    MTP (Multi-Token Prediction) — Training/inference trick where the model predicts multiple future tokens at once, speeding things up.  
    Embedding — Converts words/tokens into vectors (numbers) the model can work with.  
    Un-embedding — The reverse step: mapping the model’s internal vector back into tokens.  
    embed_tokens — The big lookup table of embeddings (token → vector).  
    shared_head.head tensors — Extra weight matrices used for prediction; they can be huge.  
    [129280, 7168] — The shape of such a tensor: ~129k rows (tokens in the vocab) × 7k columns (hidden dimension).  
    FP8 — Floating-point format using 8 bits (compact, faster, less precise).  
    Active parameters — The weights that actually need to be loaded in GPU memory to run the model.  
    Inference — Running the model to generate text (as opposed to training it).  
    GB savings — If you avoid duplicating giant matrices, you save GPU memory and speed things up.

This is my experience with Gemini. Yes, I really can put an entire codebase and all the docs and pre-dev discussions and all the inter-engineer chat logs in there.

I still see the model becoming more intoxicated as turn count gets high.

Agreed. That said, in general a 1M context model has a larger usable window than a 260k context model.

> If you read the model card, Qwen3-Next can be extended to 1M context length with YaRN.

I read the article, but as I said Qwen chat only provides up to 262k tokens in context length, so I'll stick with Qwen2.5 Turbo which supports 1M tokens.

I am not in a position where I can self-host yet

Are these actually different models vs just different names from the open weights releases?

I just compress the file size as low as possible without losing the quality, didn't even know there was more ways to prep it.

I do sometimes chop up the PDF into smaller pdfs with their own individual chapters

Not OP, but we use the docling library to extract text and put it in markdown before storing for use with an LLM.

Naturally. That's how LLMs work. During training you measure the loss, the difference between the model output and the ground-truth and try to minimize it. We prize models for their ability to learn. Here we can see that the large model does a great job at learning to draw bob, while the small model performs poorly.

For the model to have memorized the entire sequence of characters precisely, this must appear hundreds of times in the training data?

Conveniently removed the artist's signature though.

For ascii to look right, not messed up, the generator has to know the width of the div in ascii characters, e.g. 80, 240, etc, so it can make sure the lines don't wrap. So how does an LLM know anything about the UI it's serving? Is it just luck? what if you ask it to draw something that like 16:9 in aspect ratio... would it know to scale it dowm so lines won't wrap? how about loss of details if it does? Also, is it as good with Unicode art? So many questions.

Strong disagree.

Llama 4's release in 2025 is (deservedly) panned, but Llama 3.1 405b does not deserve that slander.

https://artificialanalysis.ai/#frontier-language-model-intel...

Do not compare 2024 models to the current cutting edge. At the time, Llama 3.1 405b was the very first open source (open weights) model to come close to the closed source cutting edge. It was very very close in performance to GPT-4o and Claude 3.5 Sonnet.

In essence, it was Deepseek R1 before Deepseek R1.

It's not that clear. Yes, it underperforms in recent benchmarks and usecases (i.e. agentic stuff), but it is still one of the strongest open models in terms of "knowledge". Dense does have that advantage of MoE, even if it's extremely expensive to run inference on.

Check out this great exercise - https://open.substack.com/pub/outsidetext/p/how-does-a-blind...

I would venture to suggest that to read it as "Qwen made MoEs in toto || first || better than anyone else" is reductive - merely, the # of experts and #s here are quite novel (70b...inferencing only 3b!?!) - I sometimes kick around the same take, but, thought I'd stand up for this. And I know what I'm talking about, I maintain a client that wraps llama.cpp x ~20 models on inference APIs

Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users.

Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other views.

Sure but the money people are paying right now isn't that much in the grand scheme of things. OpenAI is expecting 13bn in revenue this year. AWS made over 100bn last year. So unless they pay a lot more, or they find customers outside of programmers, designers, etc who are willing to pay for the best quality, I don't see how it grows as fast as it needs to (I'm not saying it won't increase, just not at the rate expected by the data center providers)

For early adopters yes but many systems have been running as good enough without any kind of updates for a long time. For many use cases it needs to get to a point where accuracy is good enough and then it will be set and forget. I disagree with the approach but that's what you find in the wild.

The best quality you can get is at odds with the best speed you can get. There are lots of people (especially with specific use cases) who will pay for the best speed they can get that is high enough quality.

If the price of inference drops through the floor all the AI wrapper companies become instantly more valuable. Cursor is living on borrowed time because their agents suck and they're coasting on first mover advantage with weak products in general, but their position would get much better with cheap inference.

Buy the application layer near winners. When computing costs shrink, usage expands.

Assuming your question isn't rhetorical, massive Oracle Crypto Farm.

It's the original. I'll update the label to clarify.

Could you give some practical examples? I don't know what Qwen's 36T-token training set is like, so I don't know what it's overfitting to...