It's a commonly-repeated comment that early stage startups should avoid K8s at all cost. As someone who had to manage it on a baremetal infrastructure in the past, I get where that comes from - Kubernetes has been historically hard to setup, you'd need to spend a lot of time learning the concepts and how to write the YAML configs, etc.
However, hosted K8s options have improved significantly in recent years (all cloud providers have Kubernetes options that are pretty much self-managed), and I feel like with LLMs, it's become extremely easy to read & write deployment configs.
What's your thoughts about adopting K8s as infrastructure early on (say, when you have initial customer fit and a team of 5+ engineers) and standardizing around it? How early is too early? What pitfalls do you think still exist today?
At early stage the product should usually be a monolith and there are a LOT of simple ways to deploy & manage 1 thing.
Probably not an issue for you but costs will also tend to bump up quite a lot, you will be ingesting way more logs, tons more metrics etc just for the cluster itself, you may find yourself paying for more things to help manage & maintain your cluster(s) . Security add-ons can quickly get expensive.
PS. Edit for clarity.
For my own projects I use a managed Northflank cluster on my own AWS account and likewise... just a fantastic experience. Everything that Heroku could and should have been. Yes the cluster is a bit pricey to stand up both in terms of EC2 compute and management layer costs, but once it's there, it's there. And the costs scale much more nicely than shoving side projects onto Heroku.
At this stage I consider managed k8s my default go-to unless it's something so lightweight I just want to push it to Vercel and forget about it.
Same goes for all tech choices. If you already know it, you understand its pros and cons and it still seems like the simplest best option for the concrete thing you need to build right now, use it.
Otherwise, use whatever alternative tech fits that description instead!
* Kubernetes is great for a lot of things, and I think there's many use cases for it where it's the best option bar none
* Particularly once you start piling on requirements - we need logging, we need metrics, we need rolling redeployments, we need HTTPS, we need a reverse proxy, we need a load balancer, we need healthchecks. Many (not all!) of these things are what mature services want, and k8s provides a standardized way to handle them.
* K8s IS complex. I won't lie. You need someone who understands it. But I do enjoy it, and I think others do too.
* The next best alternative in my opinion (if you don't want vendor lock in) is docker-compose. It's easy to deploy locally or on a server
* If you use docker-compose, but you find yourself wanting more, migrating to k8s should be straightforward
So to answer your questions, I think you can adopt k8s whenever you feel like it, assuming you have the expertise and are willing to dedicate time to maintaining it. I use it in my home network with a 1 node "cluster". The biggest pitfalls are all related to vendor lock in - managed Redis, Azure Key Vault. Hyper specific config related to your managed k8s provider that might be tough to untangle. At the same time, you can just as easily start small with docker-compose and scale up later as needed.
* how much downtime can be tolerated during a deploy or outage? load balancing and multi-region is more $$$.
* if you have a bunch of linux nerds and an efficient app -- a nginx webserver + your app + Postgres DB and Ansible to manage a single VM with Cloudflare in front of it might be a good option. Portainer in the VM is nice if you want to go with containers.
* if you have a bunch of desktop devs, containers and build pipelines with PaaS are a good option. many are resilient and have HTTPs built in.
* the smaller your infra/devops team, the more i would leverage team knowhow and encourage PaaS offerings.
* the smaller your budget, the more creative you need to be (ec2/storage accounts as part of hosting, singular monolithic VM has relatively flat costs, what free stuff do i have on my cloud provider, etc)
The real challenge isn’t setting up the EKS cluster, but configuring everything around it: RBAC, secrets management, infrastructure-as-code, and so on. That part takes experience. If you haven’t done it before, you're likely to make decisions that will eventually come back to haunt you — not in a catastrophic way, but enough to require a painful redesign later.
P.S. If your needs are simple, consider starting with Docker Swarm. It's surprisingly low maintenance compared to Kubernetes, which has many moving parts and frequent deprecations from cloud providers. Feel free to drop me an email. I can share a custom Python tool I've written long time ago to automate the initial setup via the AWS API.