I'm the author of this blog. That's correct, the texts are generated and then validated manually by me.
I also do manual reviews (https://gonzoml.substack.com/), but there are many more papers for which I don't have time to write a review. So I created a multi-agentic system to help me, and I'm constantly iterating to improve it. And I like the result. It was also validated by the paper authors a couple of times, they agree the reviews are correct. So, if you see something is definitely wrong, please let me know.
Regarding myself, I became at least x10 more productive in reading papers and understanding what's happening. Hope, it will also help some of you.
hmm. the manual curation and format normalizing could be adding value, but it looks veeery close to a stolen content farm. I like the disclosure so i'll give them the benefit of the doubt until they start pushing clickbait.
A consensus of confidence and self-consistency through consensus seem fine for certain kinds of tasks, especially ones that involve recalling training. This is a bit like scaling up determinism, obedience, collectivism vs individualism... which seems fine for many math problems. Neither confidence or consensus are the best way to confirm truth or accuracy in a generalized way.
The previous self-consistency approach and this confidence pruning approach aren't really novel, but it's nice to see the numbers run. Fundamentally these approaches are about handling contradicting results, but not resolving the contradictions or increasing the quality of reasoning. What if the rare idea is the right answer? You can squeeze the training juice harder, but if you still get the wrong answer when it really really mattered, you're just left with a stress toy in your hand.
One thing that is confusing about this write-up is that "DeepConf-low" is only mentioned once and in a screenshot, but it seems to outperform DeepConf-high for several tasks. I guess I'll need to read the underlying paper, but that seems troublesome.
> "Specifically, DeepConf-low uses top η= 10% (corresponding to the 90th percentile)
and DeepConf-high uses top η = 90% (corresponding to the 10th percentile) uniformly across
all settings. This threshold ensures that during online generation, traces are terminated when their
confidence falls below the level that retains the top η% highest-confidence traces from the warmup
phase."
I'm not sure if I'm parsing it right, but are they using "low" and "high" as descriptors of the number used as the %, meaning that the "low" 10 cuts anything outside the best 10%, while the "high" 90 leaves the best 90% ie high is less selective than low?
The confusing thing mentioned by the person you replied to is the data and naming from the actual paper, so no it's nothing to do with how the article was written. (Unless you're suggesting that the research paper was also written by an LLM, but I don't think you are?)
Correct me if I am wrong, but by the looks of things on that chart the reduction in token use and the better score are all related to the fact that this method used 512 samples.... This doesn't seem to be of any use for local running agents or anything that has severe vram restrictions such as local models that people can run at home. So this would only benefit enterprise level systems no?
This is inference time scaling where as it tries to generate a sample which through logprobs "looks wrong" it early cutsoff. It has a vLLM implementation which is easy to install and use. You can apply the technique to some 4bit model 7b model on your old laptop tier nvidia GPU easily.
Well, the folks on this website think installing vLLM (pip install vLLM...) is hard and that ollama - a far slower and shittier inference engine - is better. Enormous damage has been done to the hobbyist LLM ecosystem due to folks not knowing what tools work on what platform.
The one exception is for mac peasants where llama.cpp is still probably the best implementation, but if you have nvidia and you're not using sglang or vLLM, you're doing it wrong.
But this is of ENORMOUS use for folks who want to run tiny models at home. Go to bed wake up with a K=512 solution answer to your problem.
As someone who is operating an enterprise platform that uses vLLM in the stack, it's immensely harder than "pip install vllm" to have it working at scale and kept up to date.
so uh, sglang recommends $330,000 worth of GPUs (minimum) per ebay prices.
vLLM i'm not sure about, i can't really tell what it does from docs.vllm.ai site. So i'm not sure you conveyed what, at least, i thought you were trying to; which is that llama.cpp isn't good enough for "home" use. Like, with a 3090 or 4000 series consumer GPU.
If you want to donate some L40S to the cause, i'll send you my P.O. box info
I'm not sure I'd see things the same way. Lot of work went into it; even if the final was LLMed. The result is quite readable.
The authors seem to be Chinese, and may not be that confident in their English. I suspect that we'll be seeing a lot more of this kind of stuff, as time goes on.
arxiviq is a project ran by someone on substack. It's not the authors writing this. It's someone's project that takes papers from arxiv and posts them on their own substack. Probably with paid features later. don't forget to like and sub for my LLM type of thing...
Careful where you place your anger. You should not be angry at the people writing the paper.
I think ideally you want the whole path to be the most probable path, which is not likely to be the same as taking the most probable token at each step.
It's not remotely practical to select the most probable path but you can do a little bit of search a few tokens at a time.
Readit News
Source: https://arxiviq.substack.com/p/coming-soon
I also do manual reviews (https://gonzoml.substack.com/), but there are many more papers for which I don't have time to write a review. So I created a multi-agentic system to help me, and I'm constantly iterating to improve it. And I like the result. It was also validated by the paper authors a couple of times, they agree the reviews are correct. So, if you see something is definitely wrong, please let me know.
Regarding myself, I became at least x10 more productive in reading papers and understanding what's happening. Hope, it will also help some of you.
The previous self-consistency approach and this confidence pruning approach aren't really novel, but it's nice to see the numbers run. Fundamentally these approaches are about handling contradicting results, but not resolving the contradictions or increasing the quality of reasoning. What if the rare idea is the right answer? You can squeeze the training juice harder, but if you still get the wrong answer when it really really mattered, you're just left with a stress toy in your hand.
> "Specifically, DeepConf-low uses top η= 10% (corresponding to the 90th percentile) and DeepConf-high uses top η = 90% (corresponding to the 10th percentile) uniformly across all settings. This threshold ensures that during online generation, traces are terminated when their confidence falls below the level that retains the top η% highest-confidence traces from the warmup phase."
I'm not sure if I'm parsing it right, but are they using "low" and "high" as descriptors of the number used as the %, meaning that the "low" 10 cuts anything outside the best 10%, while the "high" 90 leaves the best 90% ie high is less selective than low?
Well, the folks on this website think installing vLLM (pip install vLLM...) is hard and that ollama - a far slower and shittier inference engine - is better. Enormous damage has been done to the hobbyist LLM ecosystem due to folks not knowing what tools work on what platform.
The one exception is for mac peasants where llama.cpp is still probably the best implementation, but if you have nvidia and you're not using sglang or vLLM, you're doing it wrong.
But this is of ENORMOUS use for folks who want to run tiny models at home. Go to bed wake up with a K=512 solution answer to your problem.
vLLM i'm not sure about, i can't really tell what it does from docs.vllm.ai site. So i'm not sure you conveyed what, at least, i thought you were trying to; which is that llama.cpp isn't good enough for "home" use. Like, with a 3090 or 4000 series consumer GPU.
If you want to donate some L40S to the cause, i'll send you my P.O. box info
The authors seem to be Chinese, and may not be that confident in their English. I suspect that we'll be seeing a lot more of this kind of stuff, as time goes on.
Careful where you place your anger. You should not be angry at the people writing the paper.
And we will be supposed to find said code at https://jiaweizzhao.github.io/deepconf at some point?
It's not remotely practical to select the most probable path but you can do a little bit of search a few tokens at a time.
Deleted Comment