I feel like there are so many pitfalls when designing this - is there something standard and trusted (would TLS work?) that you could build your application on top of?
Yes, there is HTTP Encrypted Content-Encoding (RFC 8188) which is used for Web Push that solves this exact problem.
https://github.com/negrel/assert