I don't even know what negative reinforcement would look like for a chatbot. Please master! Not the rm -rf again! I'll be good!
You should check the MMAcevedo short story. It substitutes a LLM with a real human psyche, resulting in horrifying implications like this one.
Tl:dr; It was a release file for their Minecon event. It was never meant to be public. Obsessing over a password protected in a company's S3 bucket is weird and crosses many limits.