Readit NewsI think right now we're at the point where novelcrafter is an excellent proxy for the best models for readers, because LLMs are still mostly losing engagement due to technical errors as opposed to subjective ones:
That's repetition problems, moralizing/soft-censorship, grammatical quirks, missing instructions, forgetting major plot points, etc.
Those kinds of errors are so obvious you can almost rank these models with an N=1 vibe test, and they limit how much people will consume unless you're scratching certain itches like NSFW
-
However I do think with enough post-training you can beat that level of problems and move to a stage where the writing is technically sound (and that's what I've spent most of the last year working on).
From there you get to more challenging problems that require much more feedback along some level of specialization per user (like what Midjourney does during onboarding to build up a style profile). Once you're not making technical mistakes, you now have to codify the ethereal concept of "user taste", and that will be a really interesting challenge for LLMs.
Really interested in what you've been working on for the past year! Are you doing custom fine-tuning or more on the prompting/post-processing side? Also I definitely need to check out the Midjourney onboarding, it sounds super interesting for inspo regarding your point about personalization + taste!
You should explore high temperature (far above 2) sampling with good truncations like min_p, top n sigma, TFS, mirostat, typicality sampling, etc. Basically anything that isn't top_p/top_k. This is the path to highly diverse outputs.