The odd thing about having 20 years of experience (while simultaneously being wide-eyed about new tech), is that I now have enough confidence to read interesting posts (like any post on k8s) and not think "I HAVE to be doing this" – and rather think "good to know when I do need it."
Even for the highest scale app I've worked on (which was something like 20 requests per second, not silicon valley insane but more than average), we got by perfectly fine with 3 web servers behind a load balancer, hooked up to a hot-failover RDS instance. And we had 100% uptime in 3 years.
I feel things like Packer (allowing for deterministic construction of your server) and Terraform are a lot more necessary at any scale for generally good hygiene and disaster recovery.
I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.
The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going.
There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it.
I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services.
I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
> I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary.
To be fair, half of the API Gateways and edge router projects out there are basically nginx with a custom consul-like service bolted on.
You can do get around the nginx Plus requirement by using a module like ngx_mruby to customize the backend selection. I haven't measured the latency, so it may not be suitable for your 10ms example.
Here's a post I wrote on that ~4 years ago that uses an in-process cache [1]. It'd be fairly easy to add an endpoint to update it and pull data from Consul. I agree with you, it's a missed opportunity - there are alternatives, but being able to rely on a battletested server like nginx makes a difference.
It appears that if the consul client has the right permissions it can restart the nginx service after editing the configuration file. It uses the consul templating engine to generate an nginx config file.
The two main components are nerve (health checking of services + writing a "I'm here and healthy" znode into zookeeper, https://github.com/airbnb/nerve) and synapse (subscribes to subtrees in zookeeper, updates nginx/haproxy/whatever configs with backend changes, and gracefully restarts the proxy, https://github.com/airbnb/synapse).
It's fairly pluggable too if you don't want to use haproxy/nginx.
I totally agree, especially about being able to serve content out of a cache instead of files. It would simplify some of my configuration especially for static sites that point to a CDN.
I like what Caddy is doing, exposing their entire configuration through a REST interface.
I 100% agree with you, I've been using Consul for four years now to run 100s of services in 1000s of VMs across datacenters distributed globally and not once I saw the need for anything else...
Maybe I just don't have the scale to find service mesh or kubernetes interesting. Nomad however is something I am willing to give a go for stateless workflows that I would usually provision a VM running a single docker container for.
> I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.
under the load point of view, yes. absolutely. no doubt.
under the speed of action, no way. if your k8s cluster is properly managed, you can let developers do most of the operations work themselves, confined into their namespaces, touching only the kind of resources that you tell them to touch.
I personally would advise against using DNS for service discovery, it wasn't designed for that.
The few milliseconds that you get though, most likely is due to your local machine not having DNS caching configured, this is quite common in Linux. Because of that every connection triggers a request to DNS server. You can install unbound for example to do it. nscd or sssd can also be configured to do some caching.
> I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
Don't resolve DNS inline rather on every DNS update, resolve it and insert new IP addresses.
Kubernetes has complexity for a reason. It's trying to solve complex problems in a standardized and mature manner.
If you don't need those problems solved then it's not going to benefit you a whole lot.
Of course if you are using docker already and are following best practices with containers then converting to Kubernetes really isn't that hard. So if you do end up needing more problems solved then you are willing to tackle on your own then switching over is going to be on the table.
The way I think about it is if you are struggling to deploy and manage the life cycle of your applications... fail overs, rolling updates, and you think you need some sort of session management like supervisord or something like to manage a cloud of processes and you find yourself trying to install and manage applications and services developed by third parties....
Then probably looking at Kubernetes is a good idea. Let K8s be your session manager, etc.
I would qualify that a little more. If you are using docker and deploying to a cloud environment already, then moving to a cloud-managed kubernetes cluster really isn't that hard.
I've seen too many full-time employees eaten up by underestimating what it takes to deploy and maintain a kubernetes cluster. Their time would have been far better spent on other things.
There's always more than one way to do things, and it's good to be aware of the trade-offs that different solutions provide. I've worked with systems like you describe in the past, and in my experience you always end up needing more complexity than you might think. First you need to learn Packer, or Terraform, or Salt, or Ansible - how do you pick one? How do you track changes to server configurations and manage server access? How do you do a rolling deploy of a new version - custom SSH scripts, or Fabric/Capistrano, or...? What about rolling back, or doing canary deployments, or...? How do you ensure that dev and CI environments are similar to production so that you don't run into errors from missing/incompatible C dependencies when you deploy? And so on.
K8s for us provides a nice, well-documented abstraction over these problems. For sure, there was definitely a learning curve and non-trivial setup time. Could we have done everything without it? Perhaps. But it has had its benefits - for example, being able to spin up new isolated testing environments within a few minutes with just a few lines of code.
> First you need to learn Packer, or Terraform, or Salt, or Ansible - how do you pick one?
You don't. These are complementary tools.
Packer builds images. Salt, Ansible, Puppet or Chef _could_ be used as part of this process, but so can shell scripts (and given the immutability of images in modern workflows, they are the best option).
Terraform can be used to deploy images as virtual machines, along with the supporting resources in the target deployment environment.
Yes. Three web servers and a load balancer is fine. Three web servers and a load balancer, repeated 1,000 times across the enterprise in idiosyncratic ways and from scratch each time, is less fine. That’s where Kubernetes-shaped solutions (like Mesos that came before it) become appropriate.
You can get a lot done with a sailboat. For certain kinds of problems you might genuinely need an aircraft carrier. But then you’d better have a navy. Don’t just wander onto the bridge and start pressing buttons.
However, a lot of new (or just bad) devs miss the whole Keep It Simple Stupid concept and think that they NEED Kubernetes-shaped solutions in order to "do it the right way".
Many times three web servers and a load balancer are exactly what you need.
I think anyone considering these wild setups should read about how stackowerflow is hosted on a couple of IIS servers. It’s a sobering reminder that you often don’t need the new cool.
Joel, if any, has always been super pragmatic and very realistic.
Not to misunderstand. For FogBugz they wrote a compiler/transpiler for Asp and PHP because the product had to run on customers servers - because "clients should not leave their private data at another company".
A typical PHP application that does a bit of database updating per request, gets some new data from the DB and templates it should handle 20 requests per second on a single $20/month VM. And in my experience from the last years, VMs have uptime >99.99% these days.
What made you settle on a multi-machine setup instead? Was it to reach higher uptime or were you processing very heavy computations per request?
Higher uptime and more computation, although this was mostly C# so very efficiently run code. It was an e-commerce site doing +100MM a year.
There was little to no room for error. I once introduced a bug in a commit that, in less than an hour, cost us $40,000. So it wasn't about performance.
Also this was 9 years ago. So adjust for performance characteristics from back then.
A single server isn't redundant. 3 behind a load balancer, where each is sized to handle 50% of the volume lets you take systems offline for maintenance without incurring downtime.
Heck, Raspberry Pis have more horsepower than the webservers in the cluster I ran around Y2k.
(outside of GP's reply) Generically, life is messy and unpredictable, never put all your eggs in one basket. Your cloud server is sitting on a physical hyp which will need maintenance or go down, or even something in your VM goes wrong or needs maintenance. Using a basic N+1 architecture allows for A to go down and B to keep running while you work on A - whether that's DNS, HTTP or SQL etc.
The odd thing about having 10 years of experience as a consultant is that you know when to write "Kubernetes" into a project proposal, even though everyone agrees that it'll be a sub-optimal solution.
But both you and their tech lead want to be able to write "used Kubernetes" on your CV in the future, plus future-oriented initiatives inside your contact's company tend to get more budget allocated to them. So it's a sound decision for everyone and for the health of the project to just go with whatever tech is fancy enough, but won't get into the way too badly.
Enter Kubernetes, the fashionable Docker upgrade that you won't regret too badly ;)
I worked on a transacted 20-60k messages/s system and am not sure K8S wouldn't be a hindrance there... Imagine writing Kafka using K8S and microservices.
I don't know about "lot more necessary". The images are one part of the equation especially to meet various regulations.
There is a ton to running a large scale service especially if you are the service that the people who are posting how wicked smart they are at k8 service runs on. Google found that out yesterday when they said "oh hey people expect support maybe we should charge". That is not new for grown ups.
The cloud existed before k8 and k8's creator has a far less mature cloud than AWS or Azure.
But this thread has convinced me of one thing. It's time to re-cloak and never post again because even though the community is a cut above some others at the end of the day it's still a bunch of marks and if you know the inside it is hard to bite your lip.
Who is giving you 100% uptime? All major providers (AWS, GCP, Azure, etc) all have had outages in the past 3 years. And that level of infrastructural failure doesn't care about whether or not you're using k8s.
Same. I like trying out new things, so I have a feel for what they're good for. I tried setting up Kubernetes for my home services and pretty quickly got to "nope!" As the article says, it surely makes sense at Google's scale. But it has a large cognitive and operational burden. Too large, I'd say, for most one-team projects.
These kinds of posts always focus on the complexity of running k8s, the large amount of concepts it has, the lack of a need to scale, and that there is a "wide variety of tools" that can replace it, but the advice never seems to become more concrete.
We are running a relatively small system on k8s. The cluster contains just a few nodes, a couple of which are serving web traffic and a variable number of others that are running background workers. The number of background workers is scaled up based on the amount of work to be done, then scaled down once no longer necessary. Some cronjobs trigger every once in a while.
It runs on GKE.
All of this could run on anything that runs containers, and the scaling could probably be replaced by a single beefy server. In fact, we can run all of this on a single developer machine if there is no load.
The following k8s concepts are currently visible to us developers: Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secret. The hardest one to understand is Ingress, because it is mapped to a GCE load balancer. All the rest is predictable and easy to grasp. I know k8s is a monster to run, but none of us have to deal with that part at all.
Running on GKE gives us the following things, in addition to just running it all, without any effort on our part: centralized logging, centralized monitoring with alerts, rolling deployments with easy rollbacks, automatic VM scaling, automatic VM upgrades.
How would we replace GKE in this equation? what would we have to give up? What new tools and concepts would we need to learn? How much of those would be vendor-specific?
If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested.
I'm in the same camp. I think a lot of these anti-k8s articles are written by software developers who haven't really been exposed to the world of SRE and mostly think in terms of web servers.
A few years ago I joined a startup where everything (including the db) was running on one, not-backed-up, non-reproducible, VM. In the process of "productionizing" I ran into a lot of open questions: How do we handle deploys with potentially updated system dependencies? Where should we store secrets (not the repo)? How do we manage/deploy cronjobs? How do internal services communicate? All things a dedicated SRE team managed in my previous role.
GKE offered a solution to each of those problems while allowing me to still focus on application development. There's definitely been some growing pains (prematurely trying to run our infra on ephemeral nodes) but for the most part, it's provided a solid foundation without much effort.
Exactly, all these articles seem to come from operational novices, who think in terms of 1-2 click solutions. K8s is not a 1-2 click solution, and clearly isn't designed to be; it's solving particular tough operational problems that if you don't know exist in the first place you won't really be able to evaluate these kinds of things properly.
If a group literally doesn't have the need to answer questions like the ones you posed, then OK, don't bother with these tools. But that's all that needs to be said - no need for a new article every week on it.
Agreed. I've been saying for years that if you go with Docker-Compose, or Amazon ECS, or something lower level, you are just going to end up rebuilding a shittier version of Kubernetes.
I think the real alternative is Heroku or running on VMs, but then you do not get service discovery, or a cloud agnostic API for querying running services, or automatic restarts, or rolling updates, or encrypted secrets, or automatic log aggregation, or plug-and-play monitoring, or VM scaling, or an EXCELLENT decoupled solution for deploying my applications ( keel.sh ), or liveness and readiness probes...
You do in fact get a lot of this stuff with ECS and Fargate - rolling updates, automatic restart, log aggregation, auto scaling, some discovery bits, healthchecks, Secrets Manager or Parameter Store if you want, etc.
1000%. If you take a little bit of time to learn k8s and run it on a hosted environment (e.g. EKS), it’s a fantastic solution. We are much happier with it than ECS, Elastic Beanstalk, etc.
> If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested.
Sure. @levelsio runs Nomad List (~100k MRR) all on a single Linode VPS. He uses their automated backups service, but it's a simple setup. No k8s, no Docker, just some PHP running on a Linux server.
As I understand it, he was learning to code as he built his businesses.
You're asking exactly what I've been wondering. The answer in this thread this so far has been "maybe dokku." I can totally buy that K8s is overkill, but the alternatives for small scale operators seem to require a lot more work internally than just using a K8s service.
same we use kubernet not for scale but for the environment repeatability. we can spin any branch at any time on any provider and be sure it's exactly as we have it in production down to networking and routing, to built that out of plain devops tools would require a lot more in depth knowledge and it won't port exactly one provider to another
> These kinds of posts always focus on the complexity of running k8s...
Yes but that is also the worst already that you could criticize about k8s.
Complexity is dangerous because if things are growing beyond a certain threshold X you will have side effects that nobody can predict, a very steep learning curve and therefor many people screwing up something in their (first) setups as well as maintainability nightmares.
Probably some day someone will prove me wrong but right now one of my biggest goals to improve security, reliability and people being able to contribute is reducing complexity.
After all this is what many of us do when they refactor systems.
I am sticking with the UNIX philosophy at this point and in the foreseeable future I will not have a big dev-team at my disposal as companies like Alphabet have to maintain and safe-guard all of this complexity.
From a developer perspective, k8s seems ripe for disruption.
It does a bunch of junk that is trivial to accomplish on one machine - open network ports, keep services running, log stuff, run in a jail with dropped privileges, and set proper file permissions on secret files.
The mechanisms for all of this, and for resource management, are transparent to unix developers, but in kubernetes they are not. Instead, you have to understand a architectural spaghetti torrent to write and execute “hello world”.
It used to be similar with RDBMS systems. It took months and a highly paid consultant to get a working SQL install. Then, you’d hire a team to manage the database, not because the hardware was expensive, but because you’d dropped $100k’s (in 90’s dollars) on the installation process.
Then mysql came along, and it didn’t have durability or transactions, but it let you be up and running in a few hours, and have a dynamic web page a few hours after that. If it died, you only lost a few hours or minutes of transactions, assuming somone in your organization spent an afternoon learning cron and mysqldump.
I imagine someone will get sufficiently fed up with k8s to do the same. There is clearly demand. I wish them luck.
This, kubernetes is container orchestration framework. If you know your needs then you have opportunity to make a better decision and planning. I had similar experience with ingress also I would like to add installing Kubernetes on bare metal is a pain in the neck, even with kubespray.
The names may seem daunting, but here's what they do: a Pod is one or more containers, usually 1, that run an application. A Deployment manages a number of the same pods, making sure the right number is running at all times and rolling out new versions. A Job runs a pod until it succeeds. A CronJob creates a job on a schedule. A service provides a DNS name for pods of a certain kind, automatically load balancing between multiple of them. An Ingress accepts traffic from the outside and routes it to services. A ConfigMap and a Secret are both just sets of variables to be used by other things, similar to config files but in the cluster.
It's a small service according to " web scale", but it's serving, and vital for, a good number of customers.
It's an afternoon worth of research to understand the basic concepts. Then, with the powerful and intuitive tooling you can spin up your own cluster on your computer in minutes and practice deploying containers that:
- are automatically assigned to an appropriate machine(node) based on explicit resource limits you define, enabling reliable performance
- horizontally scale (even automatically if you want!)
- can be deployed with a rolling update strategy to preserve uptime during deployments
- can rollback with swiftness and ease
- have liveness checks that restart unhealthy apps(pods) automatically and prevent bad deploys from being widely released
- abstracts away your infrastructure, allowing these exact same configs to power a cluster on-prem, in the cloud on bare metal or vms, with a hosted k8s service, or some combination of all of them
All of that functionality is unlocked with just a few lines of config or kubectl command, and there are tools that abstract this stuff to simplify it even more or automate more of it.
You definitely want some experienced people around to avoid some of the footguns and shortcuts but over the last several years I think k8s has easily proven itself as a substantial net-positive for many shops.
Really? If someone told me they were going to write all the glue code that basically gets you the same thing a UI deployment of k8s and a couple yaml files can provide, I’d walk out.
This, and the other articles like it, should be required reading on any "how to startup" list. I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time. For request loads one hundredth of what I was handling effortlessly with a monolithic Rails server in 2013.
It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. k8s on day one at a startup is like a mom and pop grocery store buying SAP. It wouldn't be acceptable in any other industry, and can be a death sentence.
When I was younger, I was much more enticed with new technologies. But as I grow older, I've grown much more cynical. I just want to solve problems with ideally as little coding/thinking as possible.
When I was younger, I believed technology is developed to solve problems people have. Today, I've grown much more cynical, and believe that most technology is developed to sell worthless crap to other people, which may or may not occasionally do something useful for them (but not as much as it would do if the vendor actually cared).
When I was younger, I was already suspicious of hyped, marketing-driven technologies and lock-in. But as I grow older, I've grown more aware at how people constantly ignore the problem.
I just want to solve problems with ideally as little complexity as possible.
Anecdata: series B startup. First year was a single VM with two docker containers for api and nginx. Deploy was a one shot “pull new container, stop old, start new” shell command. Year 2 onwards, k8s. No regrets, we only needed to make our first dedicated ops hire after 15 engineers, due in large part to the infra being so easy to program.
I used GKE and I was also very familiar with k8s ahead of time. I would not recommend someone in my shoes to learn k8s from scratch at the stage I rolled it out, but if you know it already, it’s a solid choice for the first point that you want a second instance.
When people talk about companies using k8s too soon, they are talking about deploying it on your own, not using a hosted service like GCE. That’s a whole new ball game and takes 100x the effort.
Tech lead in my team is full of common sense when it comes to not getting exited by JavaScripts framework of the season. But he loves to add new tools to the back end. He is choosing the "best tool" for the job, but we have a small team and already have way more things than we can comfortably manage. Good enough tools that are already in our system would be more pragmatic.
Hopefully that's not a case of "must be seen to be doing something" syndrome. If you can use pre-existing tools and they serve the purpose, how easy or difficult is it to resist the "shift to this" pressure?
Plain docker - hell on earth. Literally some of the worst stuff I had to deal with. A noticeable worsening vs. running the contents of the container unpacked in /opt/myapp.
Heroku, Dokku - Very depends. A dance between "Simple and works" and "Simple and Works but my Startup is bankrupt".
K8s - Do I have more than 1-2 custom deployed applications? Predictable cost model, simple enough to manage (granted I might be an aberration), easy to combine infrastructural parts with the money-making parts. Strong contender vs Heroku-like, especially on classic startup budget
What about raw systemd? It can do resource control, service monitoring and restart and seems pretty lightweight compared to Docker. Fewer ways to footgun with data being erased, no separate shells or finding a tool you need isn't available inside the container.
> I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time.
If Kubernetes had only cost us a year and two hundred thousand dollars then we'd have been luckier than we actually are.
It definitely has a place, but it is so not a good idea for a small team. You don't need K8s until you start to build a half-assed K8s.
> You don't need K8s until you start to build a half-assed K8s.
which you start doing with more than 1 db server and more than one app server.
until you realize you have a ansible script that is tied to your specific program. oh shit now you have two programs, and copied some stuff from ansible, but not everything is the same - damn.
deployments occur a downtime (5 seconds), which some user notice - until you add like a thousand lines of ansible code.
now you need monitoring and oh shit another ansible bloat, soon you outgrown k8s codebase with just ansible.
(p.s. this does not account how you start your baremetal servers, etc. this would be another story)
>It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business.
Then they might simply join another startup or a big tech company as competition for good engineers is fierce. Startups also famously underpay versus larger companies so you need to entice engineers with something.
Well, when you pay your engineers 6-12 months of extra salary before you ship anything because they had to use Kubernetes-on-Highways to host this clever NoNoNoPleaseNoSQL DB hosted that some guy on Github wrote last week, hosted on ZeroNinesAsAService.com and with a new UI built in ThreeReact (the hot new React-based framework that implements an OpenGL interface that works on approximately 3% of devices in the wild right now, and approximately 0% of your target user base's devices), don't forget to account for that in the investor pitch and salary offers.
I mean, seriously, this is a startup killer. Our host wrote an essay a long time ago about beating standard companies stuck in boring old Java or C++ with your fast, agile Python code, but in 2020 it seems to me it's almost more important now to try to convince new startups to be a little more boring. Whatever your special sauce that you're bringing to market is, it isn't (with no disrespect to the relevant communities) that you're bringing in Rust or Nim or whatever for the first time ever, for Maximum Velocity. Just use Python, cloud technologies, and established databases. Win on solving your customer needs.
While by no means is everyone in the world using effective tech stacks well-chosen to meet the needs and without over-privileging "what everyone else is doing and what has always been done", enough people are now that it's probably not a competitive advantage anymore.
Honestly, you can beat most companies in just getting stuff out the door quickly.
(Excuse me, off to file incorporation papers for ZeroNines LLC. Why wonder whether your provider will be up when you can know? Nobody else in the business can make that promise!)
If they are more focused on playing with toys or polishing their resumes for their next job, then I'd want them to leave. Their goal should be understanding what users need and building it as simply and effectively as possible.
On the flip side, a lot of developers are more motivated by what they can actually build and release with their tools, rather than learning new and complicated tools for the heck of it. If I were running a company I'd much rather court devs in this group.
This is a really good argument for why startups should build the initial product with only the founding team, and hire only once you have more demand than you can possibly handle. You get mission-alignment from your technical team (because they only get paid if they ship and customers appear), and then finding and retaining engineers is a lot easier once you're clearly a rocket-ship, their stock options double in value every 6 months, and they become a manager after 6 months in the company.
1000 users, almost 0 request per second / 10000 euro a month to deploy a very simple software on GKE in 3 continents and have some development instances.
Why? Because Docker and "scalability" it offered looked much better on the investor slides...
How? Instead of actually hiring someone that at least has experience with docker he decided it was very easy thing to learn so he did it him self and we ended up with thing like running two application in a same container, having database on containers that disappear (container restarts when ram is full is the best way to run a database) etc...
And after all that started talking about mirco-services and how cool are they for code decoupling... Of all the things... I don't work there anymore...
And if challenged on these reasons some people give (that supposedly have more experience) blanket statements like: "Docker is more safe by default thus we should use it..."
Maybe when you go thought these situations you get to write articles like this.
Of course containers, docker, k8 etc have their place, but in reality you can find all kinds of stunning non-sense.
I worked at a startup where the CTO was the impressionable junior and it caused all of those issues you described and more. I ended up leaving when I decided I shouldn't be having panic attacks in the middle of the night due to ridiculously tight deadlines that I couldn't meet because of the overhead introduced by k8s and microservices.
I've seen the pressure come from customers too, in the case of b2b between tech companies. My company embraced Kubernetes after years of customers asking us "what's your Kubernetes story?" and not liking the answer, "that doesn't solve any problems we have".
Sure, it is the job of the CTO to steer juniors in an appropriate direction. But, and I hate to say this, isn't it also the job of the CTO to help source funding and it seems VCs like to go after the buzzwords?
People managed with dynamically typed languages for a long, long time in building pretty vast services. Static typing has its merits but it really depends on what you're building - if you're a bank, it offers excellent guarantees. If you're building something smaller scale or lower impact, you can cope if you're logical and have good tests.
Having been at a company that was starting to move things to Kubernetes, when it had absolutely no reason to, I can say that it was being done because:
1) the developers wanted to be able to say they knew how to use Kubernetes, when they applied for their next job (perhaps at a company big enough to need it)
2) the managers didn't really understand much about what it was, to evaluate if it was necessary, but
3) some of the managers wanted to say they had managed teams that used Kubernetes, for the same reason as the developers
Which is not to say that it should never be used. But we have a recurring pattern of really, really large companies (like FAANG) developing technologies that make sense for them, and then it gets used at lots of other companies that will never, ever be big enough to have it pay off. On the other hand, they now need 2-3x the developers they used to, because they have too many things going on, mostly related to solving scale problems they'll never have.
Don't use a semi-tractor trailer to get your groceries. Admit it when you're not a shipping company. For most of us, the compact car is a better idea.
Well, for engineers sometimes it's just interesting to learn new tech, you know.
And managers are ok with keeping the team motivated, especially if they can deliver new features while playing with the new tech.
UPDATE:
I mean, it's not about resume in most cases I've seen. Usually, people who don't love programming do not care about what tech to use, so resume-oriented people are usually not very vocal about using new cool and shiny tech. It's nerds who are, that keeps them motivated.
Job interviews should always consider if the correct tool was used for the use case.
So, if someone used k8s where it was not needed, that experience means zilch:
1) it was the wrong tool for the job
2) it was not used at the right scale. So, if I need k8s to manage a really large deployment, someone who used k8s to manage 5 machines does not have the right experience since they never used it at my scale.
Before you get on a technical interview, an automated system, an HR person and a project manager (or similar) will have to pass on the application, which encourages putting buzzwords tech terms on a CV.
Sadly today fewer and fewer companies look at soft and engineering skills and prefer to hire people based on experience with specific tools.
> was starting to move things to Kubernetes, when it had absolutely no reason to
There is "absolutely no reason" to use a system that automatically handles blue/green deployment of your containers for free and supporting auditable and revertible deployment histories?
I'd like to hear what you consider to be operational best practices!
I think teaching devs to use k8s is generally a good thing. The patterns in k8s are very well established and a developer can learn a lot by understanding how to implement them. For basic web apps on something like GCE the time needed to get operational should be minimal.
Well-said @rossdavich... also says a lot about tech company cultures: most people are just looking for the next job. Is there anyone actually interested in creating and contributing to building great products, ones that people actually need? #rhetorical
Developers learn quickly that the only way to get a decent salary and/or job security is to cram as many buzzwords into their resume. Companies don't like to invest in training, so the only other way to get those buzzwords is to force the use of whatever cool tech into a project, whether it's appropriate or not.
And no, experience in cool buzzword tech does not count if it's a side project or open source contribution - as much as we would wish that to be the case.
We can blame developers, but they are just adapting, as humans always do, to their environment. Sometimes it is just bored devs chasing the next fad, but there is plenty of blame to be laid at the feet of modern tech companies.
So docker runs a bunch of system services but abstracts them away...
And kubernetes runs docker but abstracts that away...
and rancher runs kubernetes but abstracts that away..
Should I just wait a year for something that lets me use rancher without knowing anything about it?
I'm also a fan of Rancher. Especially the newer versions. It significantly simplifies the process of spawning up and managing a Kubernetes cluster.
I do think that Kubernetes is overkill if you just want to spawn a couple of server instances.
But if you want to build a complex system which scales and you happen to have experienced developers on the team who understand how to build scalable systems and who have experience with Kubernetes then you'd be foolish not to use K8s IMO.
That said, having that specific talent and experience on the team is critical or else you're just wasting your time and money with K8s - And that talent is very hard to find.
There is no point building some complex architecture for Kubernetes if that architecture is not designed to scale linearly from the ground up.
Kubernetes can allow you to operate highly scalable systems easily and elegantly, but simply using Kubernetes doesn't itself guarantee in any way that your systems will be scalable and will benefit from K8s at all (aside from pretty UI).
Very few people can build systems that scale and also meet business requirements.
You clearly understood one of the primary incentives that drives k8s adoption (career self-interest).
And then you issued advice like "Don't do this". I'm curious if you truly believe that anybody in the above class of people would change their ways on that basis, and without a greater incentive.
Doing a migration when you're not clear on what problem you're solving and why is always a bad idea. But I don't think this is a fair characterization of Kubernetes. There are lots of well supported hosted ways to use it, in which case it works a lot like other PAAS tools like Heroku or Elastic Beanstalk, except with much more powerful and better designed primitives and a big open source community of tools to make working with it even easier.
It's not like using a tractor trailer to get your groceries, it's more like using a swiss army knife to cut a piece of string. Sure you don't need all the extra accessories for that task, but they're not really getting in your way, it's still a good tool for the job. And the extras might even come in handy at some point in the future.
I am a solo developer (full stack, but primarily frontend), and Kubernetes has been a game changer for me. I could never run a scalable service on the cloud without Kubernetes. The alternative to Kubernetes is learning proprietary technologies like "Elastic Beanstalk" and "Azure App Service" and so on. No thank you. Kubernetes is very well designed, a pleasure to learn and a breeze to use. This article seems to be about setting up your own Kubernetes cluster. That may be hard; I don't know; I use Google Kubernetes Engine.
For others considering Kubernetes: go for it. Sometimes you learn a technology because your job requires it, sometimes you learn a technology because it is so well designed and awesome. Kubernetes was the latter for me, although it may also be the former for many people.
The first step is to learn Docker. Docker is useful in and of itself, whether you use Kubernetes or not. Once you learn Docker you can take advantage of things like deploying an app as a Docker image to Azure, on-demand Azure Container Instances and so on. Once you know Docker you will realize that all other ways of deploying applications are outmoded.
Once you know Docker it is but a small step to learn Kubernetes. If you have microservices then you need a way for services to discover each other. Kubernetes lets you use DNS to find other services. Learn about Kubernetes' Pods (one or more Containers that must reside on the same machine to work), ReplicaSets (run multiple copies of a Pod), Services (exposes a microservice internally using DNS), Deployments (lets you reliably roll out new software versions without downtime, and restarts pods if they die) and Ingress (HTTP load balancing). You may also need to learn PersistentVolumes and StatefulSets.
The awesome parts of Kubernetes include the kubectl exec command which lets you log into any container without almost any setup or password, kubectl logs to view stdout from your process, kubectl cp to copy files in and out, kubectl port-forward to make remote services appear to be running on your dev box, and so on.
> Once you know Docker you will realize that all other ways of deploying applications are outmoded.
This is a strong and absolute statement to be making in a field as broad and diverse as software engineering. My experience from being on both sides of these statements it that they're often wrong, or at least short sighted.
In this case, while I get the packaging benefits of Docker, there are other ways to package applications that don't require as much extra software/virtualization/training. So the the question isn't as much about whether Docker/K8S/etc. provides useful benefits as whether or not those benefits are worth the associated costs. Nothing is free, after all, and particularly for small to moderate sized systems, the answer is often that the costs are too high. (And with hardware as good as it is these days, small-to-moderate is an awful lot of capacity.)
I've personally gotten a lot of value out of packaging things up into an uber jar, setting up a standard install process/script, and then using the usual unix tooling (and init.d) to manage and run the thing. I guess that sounds super old fashioned, but the approach has been around a long time, is widely understood, and known to work in many, many, many worthwhile circumstances.
When something breaks you log in to the machine and make incremental updates to fix it, right? This approach leads to non-reproducible deployment environments. Immutable systems are better, and Dockerfile is essentially a written record of how to reproduce the environment.
Everybody feels confident in the stack they have spent time using.. You like Kubernetes because you took the time to learn it, someone else will find Elastic Beanstalk or AWS ECS equally easy to setup and scale. It's not that Docker is the only way to deploy an application either, there are virtues of learning the serverless deployment modes as well on the various clouds. For many of the "proprietary lock-ins" you run into, you often get something back.
I do agree in point that kubernetes and Docker are nice of course :)
Another advantage of Kubernetes over things like Elastic Beanstalk is portability. Your app can move from one cloud to another with minimal effort.
Yet another advantage is portability and durability of your knowledge. Kubernetes has so much momentum, so it is here to stay. It is extensible so third parties can innovate without leaving Kubernetes, which is yet another reason it is going to be around for a long time.
EB or ECS specific knowledge is AWS specific. I can (and do) run k8s on my laptop and can (and do) deploy Helm charts (the ones I wrote or 3rd party ones) on any k8s install. So that's quite different to the usual vendor lock in that comes with proprietary cloud services.
I have no idea how to deploy my app on Google App Engine or Heroku. So instead of spending time developing features my customers care about, I'll spend time learning how to deploy my app on those services.
I'm skeptical to believe the service is any more scalable than it would be with regular instances and multi az. mainly because in my experience scalability has way more to do with network topology and the architecture of how requests flow, rather than the tech for implementation.
It is still in development, so no rpm at the moment.
That’s another thing: some people think Kubernetes is something you use if you need high scalability. I disagree. Kubernetes should be the default if your app consists of more than 1 service. If you don’t have high scalability requirements you can rent a single-node GKE “cluster” for about $60 per month.
If you have just 1 service then a single Docker container is all you need, so Kubernetes not needed.
This is my experience too. I've used smaller-scale tools (such as docker-compose, Dokku, Heroku etc) but I've found them to be a mixture of unreliable or unsuitable in the case of fairly modest complexity.
Eventually I turned to Kuberenetes to see how it compared. I spent a day-ish reading through the 'core concepts' in the docs which was plenty enough to get me started on GKE. It took me a week or two to migrate our workloads over, and once everything stabilised it has been pretty much fire-and-forget.
I have about twenty pieces of software deployed for my current client and I feel that I can trust Kuberenetes to just get on with making everything run.
I've since deployed clusters manually (i.e. bare metal), but I certainly wouldn't recommend it for anyone starting-out. Personally I'm keeping a close eye on k3s.
I think my main learning during this process – at least for my situation – was to run any critical stateful services outside of Kubernetes (Postgres, message queues, etc). I think this applies less now than it did when I started out (v1.4), but non-the-less it is a choice that is still serving me well.
"I could never run a scalable service on the cloud without Kubernetes."
But also
"The alternative to Kubernetes is learning proprietary technologies like "Elastic Beanstalk" and "Azure App Service" and so on. No thank you"
So can we clarify that you truly meant: "I decided not to run a scalable service in the cloud using any of the existing cloud tools that do and have supported that scenario for years. And decided to use k8s instead" :)
What happens when the IP address changes? You need some way to lookup current IP addresses. Why re-invent DNS? Also, how do you protect these services from unauthorized access?
I'm also enjoying Kubernetes. I started a hobby project on GKE just to learn, but now the project has 8,000 MAU or so and will be scaling up more in the near future. K8s is totally overkill, but I've had a good time and it's worked well so far.
I run a Saas business solo, for eight years now, netting six figures, and I've been on Heroku the entire time for just under $1,000 a month. Monolithic rails app on a single database, 300 tables.
Sometimes I feel teased by 'moving to EC2' or another hot topic to save a few bucks, but the reality is I've spent at most 2 hours a month doing `heroku pg:upgrade` for maintenance once a year, and `git push production master` for deploys and I'd like to keep it that way. I just hope Heroku doesn't get complacent as they are showing signs of aging. They need a dyno refresh, http/2, and wildcard SSL out of the box. I honestly have no idea what the equivalent EC2/RDS costs are and I'm not sure I want to know.
You should look into render.com which provides a service similar to Heroku. I haven't used them myself and have no connection with them. Their name does pop up a fair bit though.
Software engineering is the perfect example of the "blind scientists and the elephant" problem. It is a very complex field, with a number of related but distinct disciplines and activities required to make it work; it's impossible to be an expert in everything, so we tend to specialise: we have back-end engineers, front-end engineers, data engineers, SRE experts, devops specialists, database experts, data scientists and so on. Additionally, the software we are building varies wildly in terms of complexity, dependencies, external requirements etc; and finally, the scale of that software and the teams building it can vary from one person to literally thousands.
Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped".
The reality is that various technologies and patterns -- microservices, monoliths, Kubernetes, Heroku, AWS, whatever -- are tools that enable us to solve certain problems in software development. And different teams have different problems and need different solutions, and each needs to carefully weigh their options and adopt the solutions that work the best for them. Yes, choosing the wrong solutions can be expensive and might take a long time to fix, but that can happen to everyone and actually shows how important it is to understand what is actually needed. And it's completely pointless to berate someone for their choices unless you have a very detailed insight into their particular needs.
> Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped".
It's my experience the opposite is true. The blindness is people overestimating their needs (or resume-padding) and using specialized, overcomplicated tools meant for traffic in the billions (e.g. cassandra, kafka, mapreduce) for 20-person startups that haven't hit rapid growth (most of which never do).
I'm afraid you might be falling into the exact trap I have described. Realistically, how many such cases have you seen? And of those you have, how many did actually implement such a complex solution and ran it for a long time without either closing down or transforming to something more suited to their needs?
I disagree with the HN consensus here: I think managed kubernetes is really useful for startups and small teams. I also commonly hear folks recommending that I use docker-compose or nomad or something: I don't want to manage a cluster, I want my cloud to do that.
We run a fairly simple monolith-y app inside kubernetes: no databases, no cache, no state: 2 deployments (db-based async jobs and webserver), an ingress (nginx), a load balancer, and several cron jobs. Every line of infrastructure is checked into our repo and code reviewed.
With k8s we get a lot for free: 0 downtime deployments, easy real time logging, easy integration with active-directory for RBAC, easy rollbacks.
Readit News
Even for the highest scale app I've worked on (which was something like 20 requests per second, not silicon valley insane but more than average), we got by perfectly fine with 3 web servers behind a load balancer, hooked up to a hot-failover RDS instance. And we had 100% uptime in 3 years.
I feel things like Packer (allowing for deterministic construction of your server) and Terraform are a lot more necessary at any scale for generally good hygiene and disaster recovery.
The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going.
There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it.
I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services.
I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
To be fair, half of the API Gateways and edge router projects out there are basically nginx with a custom consul-like service bolted on.
Here's a post I wrote on that ~4 years ago that uses an in-process cache [1]. It'd be fairly easy to add an endpoint to update it and pull data from Consul. I agree with you, it's a missed opportunity - there are alternatives, but being able to rely on a battletested server like nginx makes a difference.
[1] http://hokstadconsulting.com/nginx/mruby-virtualhosts
https://learn.hashicorp.com/consul/integrations/nginx-consul...
It appears that if the consul client has the right permissions it can restart the nginx service after editing the configuration file. It uses the consul templating engine to generate an nginx config file.
I haven't tried it myself but it looks promising.
Airbnb's Smartstack works well for this. It's not built in to nginx as a module, but I think it's more composeable this way.
Blog post: https://medium.com/airbnb-engineering/smartstack-service-dis...
The two main components are nerve (health checking of services + writing a "I'm here and healthy" znode into zookeeper, https://github.com/airbnb/nerve) and synapse (subscribes to subtrees in zookeeper, updates nginx/haproxy/whatever configs with backend changes, and gracefully restarts the proxy, https://github.com/airbnb/synapse).
It's fairly pluggable too if you don't want to use haproxy/nginx.
I like what Caddy is doing, exposing their entire configuration through a REST interface.
I 100% agree with you, I've been using Consul for four years now to run 100s of services in 1000s of VMs across datacenters distributed globally and not once I saw the need for anything else...
Maybe I just don't have the scale to find service mesh or kubernetes interesting. Nomad however is something I am willing to give a go for stateless workflows that I would usually provision a VM running a single docker container for.
under the load point of view, yes. absolutely. no doubt.
under the speed of action, no way. if your k8s cluster is properly managed, you can let developers do most of the operations work themselves, confined into their namespaces, touching only the kind of resources that you tell them to touch.
The few milliseconds that you get though, most likely is due to your local machine not having DNS caching configured, this is quite common in Linux. Because of that every connection triggers a request to DNS server. You can install unbound for example to do it. nscd or sssd can also be configured to do some caching.
Don't resolve DNS inline rather on every DNS update, resolve it and insert new IP addresses.
If you don't need those problems solved then it's not going to benefit you a whole lot.
Of course if you are using docker already and are following best practices with containers then converting to Kubernetes really isn't that hard. So if you do end up needing more problems solved then you are willing to tackle on your own then switching over is going to be on the table.
The way I think about it is if you are struggling to deploy and manage the life cycle of your applications... fail overs, rolling updates, and you think you need some sort of session management like supervisord or something like to manage a cloud of processes and you find yourself trying to install and manage applications and services developed by third parties....
Then probably looking at Kubernetes is a good idea. Let K8s be your session manager, etc.
I've seen too many full-time employees eaten up by underestimating what it takes to deploy and maintain a kubernetes cluster. Their time would have been far better spent on other things.
K8s for us provides a nice, well-documented abstraction over these problems. For sure, there was definitely a learning curve and non-trivial setup time. Could we have done everything without it? Perhaps. But it has had its benefits - for example, being able to spin up new isolated testing environments within a few minutes with just a few lines of code.
You don't. These are complementary tools.
Packer builds images. Salt, Ansible, Puppet or Chef _could_ be used as part of this process, but so can shell scripts (and given the immutability of images in modern workflows, they are the best option).
Terraform can be used to deploy images as virtual machines, along with the supporting resources in the target deployment environment.
You can get a lot done with a sailboat. For certain kinds of problems you might genuinely need an aircraft carrier. But then you’d better have a navy. Don’t just wander onto the bridge and start pressing buttons.
However, a lot of new (or just bad) devs miss the whole Keep It Simple Stupid concept and think that they NEED Kubernetes-shaped solutions in order to "do it the right way".
Many times three web servers and a load balancer are exactly what you need.
Not to misunderstand. For FogBugz they wrote a compiler/transpiler for Asp and PHP because the product had to run on customers servers - because "clients should not leave their private data at another company".
Google it, great read.
Apparently in 2019 stack overflow was hosted in at least 25 servers, including 4 servers dedicated to run haproxy.
https://meta.stackexchange.com/questions/10369/which-tools-a...
What made you settle on a multi-machine setup instead? Was it to reach higher uptime or were you processing very heavy computations per request?
There was little to no room for error. I once introduced a bug in a commit that, in less than an hour, cost us $40,000. So it wasn't about performance.
Also this was 9 years ago. So adjust for performance characteristics from back then.
Heck, Raspberry Pis have more horsepower than the webservers in the cluster I ran around Y2k.
Serving static files with Elixir/Phoenix has a performance of 300 requests per second.
Python+gunicorn serves about 100 requests per second of JSON from postgres data.
But both you and their tech lead want to be able to write "used Kubernetes" on your CV in the future, plus future-oriented initiatives inside your contact's company tend to get more budget allocated to them. So it's a sound decision for everyone and for the health of the project to just go with whatever tech is fancy enough, but won't get into the way too badly.
Enter Kubernetes, the fashionable Docker upgrade that you won't regret too badly ;)
The cloud existed before k8 and k8's creator has a far less mature cloud than AWS or Azure.
But this thread has convinced me of one thing. It's time to re-cloak and never post again because even though the community is a cut above some others at the end of the day it's still a bunch of marks and if you know the inside it is hard to bite your lip.
It is very rare to have a complete region outage so it is pretty close to 100% uptime.
Kubernetes is not for you. 5kQPS times a hundred or more services and Kubernetes fits the bill.
> And we had 100% uptime in 3 years.
Not a single request failed in that time serving at 20 QPS? I'm a little suspicious.
Regardless, if you were handling 10 or 100 times this volume to a single service, you'd want additional systems in place to assure hitless deploys.
Things that aren't monitored are also things that don't fail.
Dead Comment
We are running a relatively small system on k8s. The cluster contains just a few nodes, a couple of which are serving web traffic and a variable number of others that are running background workers. The number of background workers is scaled up based on the amount of work to be done, then scaled down once no longer necessary. Some cronjobs trigger every once in a while.
It runs on GKE.
All of this could run on anything that runs containers, and the scaling could probably be replaced by a single beefy server. In fact, we can run all of this on a single developer machine if there is no load.
The following k8s concepts are currently visible to us developers: Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secret. The hardest one to understand is Ingress, because it is mapped to a GCE load balancer. All the rest is predictable and easy to grasp. I know k8s is a monster to run, but none of us have to deal with that part at all.
Running on GKE gives us the following things, in addition to just running it all, without any effort on our part: centralized logging, centralized monitoring with alerts, rolling deployments with easy rollbacks, automatic VM scaling, automatic VM upgrades.
How would we replace GKE in this equation? what would we have to give up? What new tools and concepts would we need to learn? How much of those would be vendor-specific?
If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested.
A few years ago I joined a startup where everything (including the db) was running on one, not-backed-up, non-reproducible, VM. In the process of "productionizing" I ran into a lot of open questions: How do we handle deploys with potentially updated system dependencies? Where should we store secrets (not the repo)? How do we manage/deploy cronjobs? How do internal services communicate? All things a dedicated SRE team managed in my previous role.
GKE offered a solution to each of those problems while allowing me to still focus on application development. There's definitely been some growing pains (prematurely trying to run our infra on ephemeral nodes) but for the most part, it's provided a solid foundation without much effort.
If a group literally doesn't have the need to answer questions like the ones you posed, then OK, don't bother with these tools. But that's all that needs to be said - no need for a new article every week on it.
Think again. There's plenty of SREs at FAANGs that dislike the unnecessary complexity of k8s, docker and most "hip" devops stuff.
I think the real alternative is Heroku or running on VMs, but then you do not get service discovery, or a cloud agnostic API for querying running services, or automatic restarts, or rolling updates, or encrypted secrets, or automatic log aggregation, or plug-and-play monitoring, or VM scaling, or an EXCELLENT decoupled solution for deploying my applications ( keel.sh ), or liveness and readiness probes...
But nobody needs those things right?
Eg. Sidecar pattern resolves most things (eg. logging)
Sure. @levelsio runs Nomad List (~100k MRR) all on a single Linode VPS. He uses their automated backups service, but it's a simple setup. No k8s, no Docker, just some PHP running on a Linux server.
As I understand it, he was learning to code as he built his businesses.
https://twitter.com/levelsio/status/1177562806192730113
Yes but that is also the worst already that you could criticize about k8s.
Complexity is dangerous because if things are growing beyond a certain threshold X you will have side effects that nobody can predict, a very steep learning curve and therefor many people screwing up something in their (first) setups as well as maintainability nightmares.
Probably some day someone will prove me wrong but right now one of my biggest goals to improve security, reliability and people being able to contribute is reducing complexity.
After all this is what many of us do when they refactor systems.
I am sticking with the UNIX philosophy at this point and in the foreseeable future I will not have a big dev-team at my disposal as companies like Alphabet have to maintain and safe-guard all of this complexity.
It does a bunch of junk that is trivial to accomplish on one machine - open network ports, keep services running, log stuff, run in a jail with dropped privileges, and set proper file permissions on secret files.
The mechanisms for all of this, and for resource management, are transparent to unix developers, but in kubernetes they are not. Instead, you have to understand a architectural spaghetti torrent to write and execute “hello world”.
It used to be similar with RDBMS systems. It took months and a highly paid consultant to get a working SQL install. Then, you’d hire a team to manage the database, not because the hardware was expensive, but because you’d dropped $100k’s (in 90’s dollars) on the installation process.
Then mysql came along, and it didn’t have durability or transactions, but it let you be up and running in a few hours, and have a dynamic web page a few hours after that. If it died, you only lost a few hours or minutes of transactions, assuming somone in your organization spent an afternoon learning cron and mysqldump.
I imagine someone will get sufficiently fed up with k8s to do the same. There is clearly demand. I wish them luck.
Here is a comparison with other frameworks, from 2018: https://arxiv.org/pdf/2002.02806.pdf
Period.
Wow as a new developer coming onboard your company, I will walk out the door after seeing that, and the fact that you admit its a small serivce.
It's a small service according to " web scale", but it's serving, and vital for, a good number of customers.
- are automatically assigned to an appropriate machine(node) based on explicit resource limits you define, enabling reliable performance
- horizontally scale (even automatically if you want!)
- can be deployed with a rolling update strategy to preserve uptime during deployments
- can rollback with swiftness and ease
- have liveness checks that restart unhealthy apps(pods) automatically and prevent bad deploys from being widely released
- abstracts away your infrastructure, allowing these exact same configs to power a cluster on-prem, in the cloud on bare metal or vms, with a hosted k8s service, or some combination of all of them
All of that functionality is unlocked with just a few lines of config or kubectl command, and there are tools that abstract this stuff to simplify it even more or automate more of it.
You definitely want some experienced people around to avoid some of the footguns and shortcuts but over the last several years I think k8s has easily proven itself as a substantial net-positive for many shops.
It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. k8s on day one at a startup is like a mom and pop grocery store buying SAP. It wouldn't be acceptable in any other industry, and can be a death sentence.
I just want to solve problems with ideally as little complexity as possible.
Nothing cynical about it.
Deleted Comment
I used GKE and I was also very familiar with k8s ahead of time. I would not recommend someone in my shoes to learn k8s from scratch at the stage I rolled it out, but if you know it already, it’s a solid choice for the first point that you want a second instance.
Tech lead in my team is full of common sense when it comes to not getting exited by JavaScripts framework of the season. But he loves to add new tools to the back end. He is choosing the "best tool" for the job, but we have a small team and already have way more things than we can comfortably manage. Good enough tools that are already in our system would be more pragmatic.
Dead Comment
Building with 12-factor principles make that transition effortless when the time comes.
Plain docker - hell on earth. Literally some of the worst stuff I had to deal with. A noticeable worsening vs. running the contents of the container unpacked in /opt/myapp.
Heroku, Dokku - Very depends. A dance between "Simple and works" and "Simple and Works but my Startup is bankrupt".
K8s - Do I have more than 1-2 custom deployed applications? Predictable cost model, simple enough to manage (granted I might be an aberration), easy to combine infrastructural parts with the money-making parts. Strong contender vs Heroku-like, especially on classic startup budget
If Kubernetes had only cost us a year and two hundred thousand dollars then we'd have been luckier than we actually are.
It definitely has a place, but it is so not a good idea for a small team. You don't need K8s until you start to build a half-assed K8s.
which you start doing with more than 1 db server and more than one app server. until you realize you have a ansible script that is tied to your specific program. oh shit now you have two programs, and copied some stuff from ansible, but not everything is the same - damn. deployments occur a downtime (5 seconds), which some user notice - until you add like a thousand lines of ansible code. now you need monitoring and oh shit another ansible bloat, soon you outgrown k8s codebase with just ansible. (p.s. this does not account how you start your baremetal servers, etc. this would be another story)
Then they might simply join another startup or a big tech company as competition for good engineers is fierce. Startups also famously underpay versus larger companies so you need to entice engineers with something.
I mean, seriously, this is a startup killer. Our host wrote an essay a long time ago about beating standard companies stuck in boring old Java or C++ with your fast, agile Python code, but in 2020 it seems to me it's almost more important now to try to convince new startups to be a little more boring. Whatever your special sauce that you're bringing to market is, it isn't (with no disrespect to the relevant communities) that you're bringing in Rust or Nim or whatever for the first time ever, for Maximum Velocity. Just use Python, cloud technologies, and established databases. Win on solving your customer needs.
While by no means is everyone in the world using effective tech stacks well-chosen to meet the needs and without over-privileging "what everyone else is doing and what has always been done", enough people are now that it's probably not a competitive advantage anymore.
Honestly, you can beat most companies in just getting stuff out the door quickly.
(Excuse me, off to file incorporation papers for ZeroNines LLC. Why wonder whether your provider will be up when you can know? Nobody else in the business can make that promise!)
Why? Because Docker and "scalability" it offered looked much better on the investor slides...
How? Instead of actually hiring someone that at least has experience with docker he decided it was very easy thing to learn so he did it him self and we ended up with thing like running two application in a same container, having database on containers that disappear (container restarts when ram is full is the best way to run a database) etc...
And after all that started talking about mirco-services and how cool are they for code decoupling... Of all the things... I don't work there anymore...
And if challenged on these reasons some people give (that supposedly have more experience) blanket statements like: "Docker is more safe by default thus we should use it..."
Maybe when you go thought these situations you get to write articles like this.
Of course containers, docker, k8 etc have their place, but in reality you can find all kinds of stunning non-sense.
Sometimes it's the CTO who is the excitable one pursuing new hotness...
I can’t understand for the life of me why any start up uses it, it’s insane.
Which is not to say that it should never be used. But we have a recurring pattern of really, really large companies (like FAANG) developing technologies that make sense for them, and then it gets used at lots of other companies that will never, ever be big enough to have it pay off. On the other hand, they now need 2-3x the developers they used to, because they have too many things going on, mostly related to solving scale problems they'll never have.
Don't use a semi-tractor trailer to get your groceries. Admit it when you're not a shipping company. For most of us, the compact car is a better idea.
And managers are ok with keeping the team motivated, especially if they can deliver new features while playing with the new tech.
UPDATE: I mean, it's not about resume in most cases I've seen. Usually, people who don't love programming do not care about what tech to use, so resume-oriented people are usually not very vocal about using new cool and shiny tech. It's nerds who are, that keeps them motivated.
So, if someone used k8s where it was not needed, that experience means zilch:
1) it was the wrong tool for the job
2) it was not used at the right scale. So, if I need k8s to manage a really large deployment, someone who used k8s to manage 5 machines does not have the right experience since they never used it at my scale.
Sadly today fewer and fewer companies look at soft and engineering skills and prefer to hire people based on experience with specific tools.
There is "absolutely no reason" to use a system that automatically handles blue/green deployment of your containers for free and supporting auditable and revertible deployment histories?
I'd like to hear what you consider to be operational best practices!
And no, experience in cool buzzword tech does not count if it's a side project or open source contribution - as much as we would wish that to be the case.
We can blame developers, but they are just adapting, as humans always do, to their environment. Sometimes it is just bored devs chasing the next fad, but there is plenty of blame to be laid at the feet of modern tech companies.
Should I just wait a year for something that lets me use rancher without knowing anything about it?
I do think that Kubernetes is overkill if you just want to spawn a couple of server instances. But if you want to build a complex system which scales and you happen to have experienced developers on the team who understand how to build scalable systems and who have experience with Kubernetes then you'd be foolish not to use K8s IMO.
That said, having that specific talent and experience on the team is critical or else you're just wasting your time and money with K8s - And that talent is very hard to find. There is no point building some complex architecture for Kubernetes if that architecture is not designed to scale linearly from the ground up.
Kubernetes can allow you to operate highly scalable systems easily and elegantly, but simply using Kubernetes doesn't itself guarantee in any way that your systems will be scalable and will benefit from K8s at all (aside from pretty UI). Very few people can build systems that scale and also meet business requirements.
And then you issued advice like "Don't do this". I'm curious if you truly believe that anybody in the above class of people would change their ways on that basis, and without a greater incentive.
It's not like using a tractor trailer to get your groceries, it's more like using a swiss army knife to cut a piece of string. Sure you don't need all the extra accessories for that task, but they're not really getting in your way, it's still a good tool for the job. And the extras might even come in handy at some point in the future.
Dead Comment
For others considering Kubernetes: go for it. Sometimes you learn a technology because your job requires it, sometimes you learn a technology because it is so well designed and awesome. Kubernetes was the latter for me, although it may also be the former for many people.
The first step is to learn Docker. Docker is useful in and of itself, whether you use Kubernetes or not. Once you learn Docker you can take advantage of things like deploying an app as a Docker image to Azure, on-demand Azure Container Instances and so on. Once you know Docker you will realize that all other ways of deploying applications are outmoded.
Once you know Docker it is but a small step to learn Kubernetes. If you have microservices then you need a way for services to discover each other. Kubernetes lets you use DNS to find other services. Learn about Kubernetes' Pods (one or more Containers that must reside on the same machine to work), ReplicaSets (run multiple copies of a Pod), Services (exposes a microservice internally using DNS), Deployments (lets you reliably roll out new software versions without downtime, and restarts pods if they die) and Ingress (HTTP load balancing). You may also need to learn PersistentVolumes and StatefulSets.
The awesome parts of Kubernetes include the kubectl exec command which lets you log into any container without almost any setup or password, kubectl logs to view stdout from your process, kubectl cp to copy files in and out, kubectl port-forward to make remote services appear to be running on your dev box, and so on.
This is a strong and absolute statement to be making in a field as broad and diverse as software engineering. My experience from being on both sides of these statements it that they're often wrong, or at least short sighted.
In this case, while I get the packaging benefits of Docker, there are other ways to package applications that don't require as much extra software/virtualization/training. So the the question isn't as much about whether Docker/K8S/etc. provides useful benefits as whether or not those benefits are worth the associated costs. Nothing is free, after all, and particularly for small to moderate sized systems, the answer is often that the costs are too high. (And with hardware as good as it is these days, small-to-moderate is an awful lot of capacity.)
I've personally gotten a lot of value out of packaging things up into an uber jar, setting up a standard install process/script, and then using the usual unix tooling (and init.d) to manage and run the thing. I guess that sounds super old fashioned, but the approach has been around a long time, is widely understood, and known to work in many, many, many worthwhile circumstances.
I do agree in point that kubernetes and Docker are nice of course :)
Yet another advantage is portability and durability of your knowledge. Kubernetes has so much momentum, so it is here to stay. It is extensible so third parties can innovate without leaving Kubernetes, which is yet another reason it is going to be around for a long time.
Can you give us an indication of the scale of your app? e.g rpm.
That’s another thing: some people think Kubernetes is something you use if you need high scalability. I disagree. Kubernetes should be the default if your app consists of more than 1 service. If you don’t have high scalability requirements you can rent a single-node GKE “cluster” for about $60 per month.
If you have just 1 service then a single Docker container is all you need, so Kubernetes not needed.
Eventually I turned to Kuberenetes to see how it compared. I spent a day-ish reading through the 'core concepts' in the docs which was plenty enough to get me started on GKE. It took me a week or two to migrate our workloads over, and once everything stabilised it has been pretty much fire-and-forget.
I have about twenty pieces of software deployed for my current client and I feel that I can trust Kuberenetes to just get on with making everything run.
I've since deployed clusters manually (i.e. bare metal), but I certainly wouldn't recommend it for anyone starting-out. Personally I'm keeping a close eye on k3s.
I think my main learning during this process – at least for my situation – was to run any critical stateful services outside of Kubernetes (Postgres, message queues, etc). I think this applies less now than it did when I started out (v1.4), but non-the-less it is a choice that is still serving me well.
But also
"The alternative to Kubernetes is learning proprietary technologies like "Elastic Beanstalk" and "Azure App Service" and so on. No thank you"
So can we clarify that you truly meant: "I decided not to run a scalable service in the cloud using any of the existing cloud tools that do and have supported that scenario for years. And decided to use k8s instead" :)
I find this statement quite bizarre.
Doesn't make any kind of judgement, just stating their personal fact.
I could never make a souffle without a recipe. Do you find this statement bizarre as well?
It’s worth noting that kubernetes uses containers which can be created via docket but is not dependent on docker
The official documentation provides a super simple tutorial, and then nothing. There's not even documentation of the primary config file. Frustrating.
https://github.com/kubernetes/website/issues/19139
Deleted Comment
Deleted Comment
Sometimes I feel teased by 'moving to EC2' or another hot topic to save a few bucks, but the reality is I've spent at most 2 hours a month doing `heroku pg:upgrade` for maintenance once a year, and `git push production master` for deploys and I'd like to keep it that way. I just hope Heroku doesn't get complacent as they are showing signs of aging. They need a dyno refresh, http/2, and wildcard SSL out of the box. I honestly have no idea what the equivalent EC2/RDS costs are and I'm not sure I want to know.
(Also, for everyone who doesn't know:
- clicking on a username takes you to that users profile
- clicking on the n minutes/hours/days ago take you to a permalink directly to that comment
)
Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped".
The reality is that various technologies and patterns -- microservices, monoliths, Kubernetes, Heroku, AWS, whatever -- are tools that enable us to solve certain problems in software development. And different teams have different problems and need different solutions, and each needs to carefully weigh their options and adopt the solutions that work the best for them. Yes, choosing the wrong solutions can be expensive and might take a long time to fix, but that can happen to everyone and actually shows how important it is to understand what is actually needed. And it's completely pointless to berate someone for their choices unless you have a very detailed insight into their particular needs.
It's my experience the opposite is true. The blindness is people overestimating their needs (or resume-padding) and using specialized, overcomplicated tools meant for traffic in the billions (e.g. cassandra, kafka, mapreduce) for 20-person startups that haven't hit rapid growth (most of which never do).
Deleted Comment
We run a fairly simple monolith-y app inside kubernetes: no databases, no cache, no state: 2 deployments (db-based async jobs and webserver), an ingress (nginx), a load balancer, and several cron jobs. Every line of infrastructure is checked into our repo and code reviewed.
With k8s we get a lot for free: 0 downtime deployments, easy real time logging, easy integration with active-directory for RBAC, easy rollbacks.