We have a few rules:

1. Read a good intro book cover-to-cover before trying to understand it.

2. Pay a cloud vendor to supply a working, managed Kubernetes cluster.

3. Prefer fewer larger clusters with namespaces (and node pools if needed) to lots of tiny clusters.

3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope really hard it continues to work.

This is enough to handle 10-50 servers with occasional spikes above 300. It's not perfect, but then again, once you have that many machines, pretty much every solution requires some occasional care and feeding.

My personal Kubernetes nightmare is having to build a cluster from scratch on bare metal.