I am on a m1 macbook air and I've been using Docker + Docker Desktop without much issue, and my use-case is a little simpler than yours (only running single containers at a time).
I'm genuinely curious to learn more about what these tools (Colima, podman) help enable. If I'm missing out on a performance boost, I'd definitely check them out.
is that the motivator for most, then?