The long context window isn't worth much and is currently creating more problems than it's worth for the bigs, with their "unlimited" use pricing models.
Let's take Claude 3's web UI as an example. We build it, and go the obvious route: we simply use as much of the context as possible, given chat history.
Well, now once you're 50-100K tokens in, the initial prefill takes forever, O(10 seconds). Now we have to display a warning whenever that is the case.
Now we're generating an extreme amount of load on GPUs for prefill, and it's extremely unlikely it's helpful. Writing code? Previous messages are likely to be ones that needed revisions. The input cost is ~$0.02 / 1000 tokens and it's not arbitrary/free, prefill is expensive and on the GPU.
Less expensive than inference, but not that much. So now we're burning ~$2 worth of GPU time for the 100K conversation. And all of the bigs use a pricing model of a flat fee per month.
Now, even our _paid_ customers have to take message limits on all our models. (this is true, Anthropic quietly introduced them end of last week)
Functionally:
Output limit is 4096 tokens, so tasks that are a map function (ex. reword Moby Dick in Zoomer), need the input split into 4096 tokens anyway.
The only use cases I've seen thus far that _legitimately_ benefit are needle in a haystack stuff, video with Gemini, or cases with huuuuuge inputs and small outputs, like, put 6.5 Harry Potter books into Gemini and get a Mermaid diagram out connecting characters.
In any case, I think a big barrier to starting things can often also just be the fear of failure or of wasting time. I spent a lot of time making electronic music as a hobby, and am probably better than most people at understanding and playing music, but for music to play a meaningful impact in my life I would need to put in a lot more work still.
On the other hand, I studied physics and mathematics far beyond the average person (getting a PhD and publishing multiple papers), but I had the support of the university and the academic environment to give me that extra push to do it.
There are so many things I would like to pursue in my free time. Building a small startup, writing a book, making YouTube videos, etc. I know that the most important thing is to just start, but the decision paralysis, and uncertainty about whether it will work out in the end can definitely be a barrier, since I can also just go out with friends and enjoy my life instead of spending time solitary.