Readit NewsXML is better for code, and for code parts in particular I enforce a cdata[[ part so there LLM is pretty free to do anything without escaping.
OpenAI API lets you do regex structured output and it's much better than JSON for code.
With OpenAI at least you can specify the cache key and they even have this in the docs:
Use the prompt_cache_key parameter consistently across requests that share common prefixes. Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.
It's an interesting equilibrium point. They want local businesses to suffer enough to pay up for ads. But also not too much that they die. A good local business that does not need to advertise because it is simply good is actually a burden to the aggregator even though it is exactly what the end users want to see.
In the past, when I was a in position to build a search engine, we took the trouble of always including organically ranked results that were genuinely good, regardless of whether we got paid or not. I felt it was a long term investment into creating real value for our end users and therefore our service.
This was not the best when it came to others (or even yourself 6 months later) reading the code. But it was great for cranking stuff out that was simply too tedious in other languages.
I did a version of this where the AI writes tools on the fly but gets to reuse them on future calls, trying to address the cost / performance issues. Migrations are challenging because they require some notion of an atomic update across the db and the tools.
This is a nice model of organically building software on the fly and even letting end users customize it on the fly.
For one, it eliminates a class of merge conflict that arises strictly from text formatting.
I always liked the idea of storing code in abstraction, especially editors supported edit-time formatting. I enjoy working on other people's code, but I don't think anybody likes the tedium of complying with style guides, especially ones that are enforced at the SCM level, which adds friction to creating even local, temporary revisions. This kind of thing would obviate that. That's why I also appreciate strict and deterministic systems like rustfmt. Unison goes a little further, which is neat but I think they're struggling getting adoption because of that, even though I'm pretty sure they've got some better tooling for working outside the whole ecosystem. These decoupled tools are probably a good way to go.
I was messing around with a file-less paradigm that would present a source tree in arbitrary ways, like just showing a individual functions, so you have the things you're working on co-located rather than switching between files. Kind of like the old VB IDE.