Looks pretty bad but considering this was impossible a couple years ago(as far as I know) it’s very impressive progress
Deleted Comment
Spoken like a true programmer. :)
But it's still probabilistic, and nine times out of ten isn't good enough.
Occasionally it will hallucinate responses like this:
{"key1": "value1", "key2": "value2" for i in range(n)}
Re-prompting with the parsing error message is usually enough to get it on the second try.
But escaping double-quotes and newline characters is less reliable. Even after giving it multiple examples, it correctly escapes only about half the time.
Re-prompting for escaping errors still yields a ~50% success rate.