Sounds more like the author just had bad experiences with non fiction? There are plenty of non fiction books which are good and where the book doesn’t just feel like filler. Some that come to mind quickly :
- bad blood (about theranos)
- SPQR
- King Leopold’s Ghost
- Lost in Math
- masters of doom
He's not saying that all non fiction is bad, just that the incentives are misaligned, and to be fair at least in my experience, there are a lot of popular non-fiction books where each chapter is repetitive, and I feel the whole thing could have been written in 2-3 chapters, if publishing a 30-page nonfiction book wasn't taboo
The one statistic mentioned in this overview where they observed a 67% drop seems like it could easily be reduced simply by editing 3.7’s system prompt.
What are folks’ theories on the version increment? Is the architecture significantly different (not talking about adding more experts to the MoE or fine tuning on 3.7’s worst failures. I consider those minor increments rather than major).
One way that it could be different is if they varied several core hyperparameters to make this a wider/deeper system but trained it on the same data or initialized inner layers to their exact 3.7 weights. And then this would “kick off” the 4 series by allowing them to continue scaling within the 4 series model architecture.