The sync between large groups used to be slow because of amount of data, but Element X and "sliding windows" were rolled out to help with it.
AFAIK, the public Matrix server used to be slow because of a heavy load (I think), but on my self-hosted instance that's not a problem at all.
And event the slower Element seems far better than Discord that I'm forced to use, where I can't even scroll history without the whole thing stuttering.