The list of things mentioned in the article to do and learn for a simple, personal project with k8s is absolutely staggering, in my opinion.
Having used it, there's a sizeable amount of further work needed which the article doesn't mention (e.g. learning how to use the pretty confusing google interface, finding the right logs and using their tools). So the overhead is really huge.
Furthermore, the whole system is slow. Want to run a SQL query against your postgres? You need to use a google cloud command that changes the firewall and ssh's you in on the machine... and this takes a couple of minutes, just enough to make me desist unless I _really_ need to run that query. Abysmal.
Finally, and this is a pet peeve against many advocacy blog posts, they just show you the happy path! Sure, _in the best of cases_ you just edit a file. In a more realistic case, you'll be stuck with a remote management system which is incredibly rich but also a really steep learning curve. Your setup is not performant? Good luck. Need to tweak or fine tune? Again, best of luck.
We've tried to adopt k8s 3-4 times at work and every single time productivity dropped significantly without having significant benefits over normal provisioning of machines. {Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!}
...which in turn is usually significantly slower than building your own home server (but that's another story!)
This. Time and again. The number of people who adopt complicated stuff like Kubernetes for what is essentially a couple of web servers and a database is too high. They're Google wannabies that thinks in Google's scale but forget that it is utterly unnecessary in their case.
I know a bio scientist who spent two months working on containers and Docker and what not, for what are essentially independent shell scripts that are best run as batch jobs. I spoke with him in length, and he realized at the end of the day that what he really needed was better understanding of standard *nix processes and not containers...
I had an excellent time working with kubernetes and I am practically a one person company. Kubernetes frees my mind from so many things that now I hate to work without it. Couple of those things include:
- automated ssl
- centralized logging
- super easy scaling up and down (going from 1 instance to 2 instance is a headache manually)
- centralized auth (have a service which doesnt have built in auth?)
- super easy recovery (containers recreated/volumes attached automatically/apps just work unless you have db type of application which shouldn't be just restarted, which is rare)
As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in.
> I know a bio scientist who spent
Super obscure example. Kubernetes is definitely not for non-tech people. And I didn't pick k8s overnight. Spent a year doing playful POCs before deploying in a production environment.
If the only thing thats stopping you from using k8s is the learning curve, I suggest you to go ahead and learn it. Its a huge boon.
I really think this kubernetes is complex meme needs to die.
I mean, sure, kubernetes _is_ complex. But it's complex in the way that using Rails is more complex than pure Ruby. It's fine when you're playing, or you have a weird subset of problems where you don't care if someone runs or not, but as soon as you deal with the problems of HA, DR, live deployments, secret management, sharing servers, working in a team, etc then if you're solving these issues with ansible and terraform, you're probably just inventing something as complex, and worse, bespoke to your problem.
At the end of the day, once you've learned to use kube, it's probably no worse than any other configuration as code based system, and the question is which of us should be learning it and when, and which of us should just be using higher levels of abstractions like gitlabs autodevops.
Now, indeed, if you're just hacking and you have no need for a CI and CD pipeline, or reproducibility, etc, then sure, it's probably not the time to learn a hosting framework, but once you do know it, you're free to make an informed opinion on your problem.
Personally, I sell kube a lot, but I tend to work for orgs that have several development teams and several apps, and bringing them to a platform seems like a good idea. The question is, should I also put a startup on GKE, or should I do something bespoke where they can't hire another contractor to take over as easily? Personally, I'd go GKE.
> This find | xargs mawk | mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation.
IMHO the point for running Kubernetes for personal projects would be to share the same virtual/physical machines between multiple projects while still keeping things manageable.
Over the years you easily end up with a pile of these projects, which want to keep running mostly for fun and therefore want to minimize the associated fixed costs. Using containers may help in keeping the herd manageable so that you can, for example, move everything to some other hosting provider.
Not that the author is right, but he does address exactly these criticims. Besides, that someone used Docker inappropriately isn't evidence that Docker is the wrong tool for personal projects (again, it may be true that Docker is the wrong tool for personal projects, but that is not evidenced by your bio scientist friend scenario).
As someone who's just starting to learn kubernetes (but has a strong background in virtualization and multi-thousand-node environments), I was inclined to agree at first but now see it as more of a modern "home lab" type person. I don't see it as doing it to be a fake-Googler, but more to learn the scalable way of doing things.
Or, as worst, the shortcut to understanding the underlying tech. But, isn't that the point of abstraction?
same here, you really have to look at your own needs. i was thinking about using something like that, but at the end of the day i would waste time and energy and complicate my setup a lot for no benefits. We are simply not at that scale, and dont have those problems, and as much as i would like to play with new technology (and i do that privately) cost/benefit calculation for us is not working right now.
> They're Google wannabies that thinks in Google's scale
and the people who, when they show up at their Google interview, nail the part about scaling. Not that Google ever asked me how to scale a project, but it doesn't hurt learning things just for the sake of learning.
When I got involved with it (and to be fair, most devops things) the value was more in what I was learning for myself, as opposed to what was suitable for the business. Because it's pretty exciting tech and everyone's talking about it, and it doesn't feel like a rabbit hole until you pull yourself back out and see the light again.
So, what happened almost every time is that the business unknowingly pivoted into innovating on PaaS, because Docker/Kubernetes was only the first step. After that, you had to figure out the best approach to CI, how to set up your deploy pipelines across multiple environments, how to replicate what you were used to seeing with Heroku, etc.
And of course the cost balloons at that stage because your cheap 3 node setup ends up exposing half a dozen load balancers that each cost money, so you start to think about hosting Traefik or nginx-ingress-controller so you can manage load balancing yourself and save some money, and because it's so easy to add more services once you know how, you start wanting to self-host all the things.
Meanwhile your startup hasn't even launched to the public yet and the sunk cost fallacy has fully embedded itself in the team mentality: they've just put months of time and effort into setting up the perfect K8S/Docker architecture and workflow, that now requires actual devops experience to maintain, and you can't push back on it because it's all about scaling when things go live, and self-hosting, and how convenient it all is.
Except, you know, that's 3-6 months of runway down the drain because leadership didn't shift the focus back to an MVP and let the fantasy continue. And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for.
> And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for.
Exactly. I was following along with the article until he mentioned that startups are doing this. Using k8s at a startup (especially a startup building an MVP) is just like using microservices. It is a disservice to the business by technologists.
Building the monolith on heroku will get you to market faster.
Yagni. (Until you do, and thrn by all means pay off that technical debt.)
I can recommend CaptainDuckDuck for simpler use cases: One-Click-Deployments (e.g. MySQL, PostgreSQL, MongoDB, ...), easy-to-use CLI tool, automatic HTTPS, Docker Swarm support (cluster and scaling), web GUI and open-source.
It made it so much easier to deploy new changes (just `captainduckduck deploy` and done). We also use minio.io (open-source S3) and are extremely productive with those tools.
It's perfect for web agencies. It's not sophisticated enough for k8s' use case, but it's extremely easy to use (you can just add nodes and it will restart containers automatically [1]).
This looks interesting. I hadn't heard of CaptainDuckDuck. Have you tried Dokku or Flynn(flynn.io)? I had looked at them and settled for Flynn because it had better support for multiple-host clusters.
Agreed 100x. I find it very hard to understand why some people talk themselves into thinking they need a distributed container orchestration and management platform, when they don’t have a container orchestration problem: they just want to run an app or two with a DB under it.
We need to go back and understand the problem we’re trying to solve, and for 99% of smallish companies and projects this is not container orchestration or hardware resource abstraction.
You don't need to run the gcloud command to connect to a Cloud SQL instance. You can run the cloud_sql_proxy as a background process on your machine and then you can use mysql or psql to connect to it instantly.
I guess your answer supports the parent's case: the happy path is a charm, but you need to learn a whole lot more to use it effectively. Paraphrasing the tag-line of Mesosphere, it's a new OS, one that is targeted at data centres, not small pet-servers setups.
I've had good experiences using Rancher (https://rancher.com/) to operate k8s clusters. It provides a nice management interface, which handles common use-cases really well. Grabbing per-container logs or getting a shell into a container becomes trivially easy.
Admittedly, operation via Rancher comes with its own complexities: setting up security groups and IAM roles. These are documented of course, but as you say: minimum effort remains non-trivial.
Can't really speak to the cloudsql issue... that's not really kubernetes. Running SQL as a container you can just kubectl port-forward.... but it sounds like a specific issue you ran into?
I'll grant that there can be headaches that I don't really see anymore because I was deeply immersed in it for a while. But running a linux server by itself was hard till I learned that too.
Performance is a subtler point, not sure I follow what you're trying to say there. Container overhead? At least in my experience gke seemed responsive, and deployments settled pretty quick.
Tweaking is usually a strong point of k8s. Just reapply... so I'm probably misunderstanding what you were getting at here.
Kubernetes needs to improve their accessibility to newcomers but the happy path isn't unheard, for simple stateless apps it mostly just works. And when it doesn't I can usually kubectl log and kubectl delete pod my way to a solution and there are docs and stack overflow to help.
Not perfect but I was surprised it worked as well as it did.
Where I work we adopted DCOS and it has been nothing but a pleasure to use and we have been updating the cluster quite often. Always very smooth and no pain at all.
We thought about k8s a couple of times but it always looked too over complicated and no gain compared to our use of DCOS
That said, once you learn all of these k8s concepts and mechanisms, and probably script nearly all of them, they all become like breathing and don't really take much time.
I frankly now am at the point where it is faster to spin up a small project on k8s than it would be to do so with the cloud providers PaaS platform.
For context: I was extremely lucky to get to work with Caleb back in 2012-13. He introduced me to Go and my life has never been the same :D
Properly hosting custom apps on the web in a resilient way isn't easy. This being the case, it seems an error to interpret the thoroughness of the article as a signal that the proposed solution or situation is unreasonably complex.
I'll take detailed and exhaustive over sparse and hand-wavy any day!
Kubernetes certainly has a learning curve, just like most worthwhile things in life.
Hosting personal projects being the task at hand, super high availability isn't usually a headline requirement.
That being said, there are certainly existing solutions for doing that, for cheap or even free tiers with less effort - Heroku, Google App Engine, etc.
It can be rough to manage Kubernetes when you just want to focus on your app. I was having that same problem when building side projects and focused on building a platform to deploy dockerfiles on a managed Kubernetes instance.
Don't forget that K8s changes rapidly so whatever you learned a year ago might be completely useless now. So better keep those people dealing with K8s learning by all means or have fun with broken production.
That is not true, based on my experience. I started to learn and use kubernetes about a year ago (now we are running our production systems on kubernetes) and we are usually on the latest version. I cannot mention anything which I learned about kubernetes and because useless, because something changed.
tl;dr if you're a team of developers sitting around a table shouting at your monitor because K8s isn't working, hire someone to do the job right for you. I think you'll find it's pretty easy and utterly worth every penny invested in it.
> Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!
Or it could mean you're not the right people for it?
Do you have a dedicated individual or team that represents infrastructure engineering? Are you a team of developers trying to play the Ops game? More often than not people/companies that struggle to get K8s going or use it are a team of developers who's expertise is code and not infrastructure/automation.
K8s is hard when done from scratch (there's a book and a video course about that by THE K8s guy.) It's not hard when you use Kops, Tectonic (discontinued, I believe), EKS, or one of the other managed services. Implementing a YAML file and launching it isn't hard. Also, you don't have to understand it inside out; you only have to understand how-to use it. Can you tell me how every component in your microwave works or do you just get on with cooking your food?
And to say you're not the use case for it - that's highly unlikely. I mean, it's possible, but it's unlikely. Your work loads and software solutions would have to be highly unique and or specialised because hey, it's not like Google's setup is simple, right?
But for personal projects, which is the context of TFA, you're not going to hire a k8s guru. As others have already said, as long as your k8s setup is working as expected, everything is fine (the happy path). But if there ever is a problem with it, then you're confronted with insane complexity, and said tools/setups on top of k8s won't fix it for you; they're even working against you as you won't have learnt the k8s foo for diagnosing wtf is the problem. And in my experience, problems occur rather sooner than later. For example, just the other week a customer of mine had their IPs exhausted on their Azure cloud. Azure and k8s will just silently fail with timeouts and all kinds of weird errors but won't tell you the root cause, and it took two experts, a fresh setup of the k8s cluster from scratch, and almost a week of downtime to fix it. It doesn't help that you only have limited diagnostics on other people's clouds either.
> and every single time productivity dropped significantly
I'm skipping job postings which require k8s. It just tells me they're not competent, falling for the hype. For AWS, ECS is way simpler and is a fully featured orchestration tool. I had an interview today, where I tried to educate why simple is better, and why they don't need k8s. No luck :)
If you design your app properly you won't ever need to run SQL queries by hand. This is a bad practice and it is good that k8s discourages such behaviour.
I agree with your other points though.
I think you are confusing the experience as a user of the app (who shouldn't really ever touch SQL) and the experience of a developer (who should be knowing SQL inside out and using it in their day-to-day work).
"If you design your app properly" applies if you're talking about using k8s to deploy an app.
If you're doing individual development, however, it's quite likely that you're also doing development of the database, and need to do ad-hoc data analysis before you include those queries in that app.
I'm beginning to think that what Kubernetes needs by far, is a better development environment story.
1. Something that makes it easy to run the application(s) you're working on locally...
2. ...but have it interact with services in a local kubernetes cluster easily.
3. Something that makes your entire development environment effectively immutable, read-from-files the same way as the immutable infrastructure we're talking so much about.
4. Something that gives you a good start to running in production, so the first day of bringing your service up isn't a mess of trying to find/remember what environment variables/config options to set.
5. Something that's one tool, instead of a mixture of `minikube`, `kubectl`, `docker`, `draft`, and `telepresence`.
6. Something that's opinionated, and follows "convention over configuration", with conventions that make for the best future designs.
Basically, we need something with a Rails-like impact but in the Kubernetes ecosystem.
Docker ships with swarm, and swarm + a compose file does everything you just described.
Swarm is such a pleasant and easy alternative that comes out of the box with Docker, does 90% of what Kubernetes does, and does 100% of what anyone with less than 50 servers needs. The learning curve is also an order of magnitude easier.
Totally agree. Docker Compose is excellent for local development, and Docker Swarm mode uses the same file and is almost the same thing. Some minor additions in extra docker-compose files and it's ready for production in a cluster. I don't get why Swarm is so underrated. For the same $5 bucks in a full Linux VPS with Linode or DigitalOcean (or even the free tier in Google Cloud) you can get a full cluster, starting with a single node, including automatic HTTPS certificates, etc. Here's my quick guide, with a Linux from scratch to a prod server in about 20 min. Including full-stack app: https://github.com/tiangolo/full-stack/blob/master/docker-sw...
Fully agree. K8s is mostly beneficial for larger companies, because of namespaces support, and devops authentication.
For smaller teams Swarm is just way more practical and versatile.
Docker (at least on my mac) now comes with kubernetes built in, so you can just click a button and instantly have a kubernetes cluster running.
As for an app deployment tool I've had good success with Helm. It allows you to define all the services, app, variables, etc. that your app needs. Though the templates do seem to be a bit convoluted and hard to read at times if you aren't careful.
I agree that we'll start to see deeper integration across the development environment though, as more dev-hours are spent on building up the toolchain.
I migrated all of my services to k8s in the last ~6 months. The biggest hurdle was the development environment (testing and deployment pipelines). I ended up with a homebrewn strategy which happens to work really well.
# Local development / testing
I use "minikube" for developing and testing each service locally. I use a micro service architecture in which each service can be tested in isolation. If all tests pass, I create a Helm Chart with a new version for the service and push it to a private Helm Repo. This allows for fast dev/test cycles.
These are the tasks that I run from a "build" script:
* install: Install the service in your minikube cluster
* delete: Delete the service from your minikube cluster
* build: Build all artifacts, docker images, helm charts, etc.
* test: Restart pods in minikube cluster and run tests.
This fits in a 200 LOC Python script. The script relies on a library though, which contains most of the code that does the heavy lifting. I use that lib for for all micro-services which I deploy to k8s.
# Testing in a dev cluster
If local testing succeeds, I proceed testing the service in a dev cluster. The dev cluster is a (temporary) clone of the production cluster, running services with a domain-prefix (e.g. dev123.foo.com, dev-peter.foo.com). You can clone data from the production cluster via volume snapshots if you need. If you have multiple people in your org, each person could spawn their own dev clusters e.g. dev-peter.foo.com, dev-sarah.foo.com.
I install the new version of the micro-service in the dev-cluster via `helm install` and start testing.
These are the steps that need automation for cloning the prod cluster:
* Register nodes and spawn clean k8s cluster.
* Create prefixed subdomains and link them to the k8s master.
* Create new storage volumes or clone those from the production cluster or somewhere else.
* Update the domains and the volume IDs and run all cluster configs.
I haven't automated all of these steps, since I don't need to spawn new dev clusters too often. It takes about 20 minutes to clone an entire cluster, including 10 minutes of waiting for the nodes to come up. I'm going to automate most of this soon.
# Deploy in prod cluster
If the above tests pass I run `helm upgrade` for the service in the production cluster.
thanks for the details, and sorry for the perhaps distracting question: how do you handle DNS for the servers on your foo.com? Did you have to provide nameservers to your registrar so K8s manages the DNS? This is something I don't see addressed so often in k8s tutorials which usually assume minikube or GCK.
Vagrant seems to be dead stupid simple, you download / git clone a folder and then you run vagrant up and follow up by vagrant ssh and look you're not on a Linux box where you can do whatever you gotta do. Using VirtualBox / Vagrant works perfectly for me so far, I'd love to give containers a shot, but if it's too much a hassle I try to avoid it.
Vagrant doesn't have to compete with containers. In fact, my best experience with Vagrant has been to standardize a development environment that then uses containers inside.
One vagrant VM allows you to make assumptions about what software is installed on a dev desktop which is especially useful if you're writing documentation. Getting into the Vagrant VM from various OSes is the main problem but once you're in... Then you can install container software and teach others how to work in docker given one assumed OS.
Since Vagrant and containers encourage some form of infrastructure/documentation as code, then more experienced developers can read through what goes into the Vagrant set up and replicate on their native OS if they prefer. As someone writing documentation, I can forget about those more experienced devs and just write to one OS.
Once you have documentation that you can guarantee works when using container software, much of the headache can be swept under the rug for more junior engineers which I think is the krux of the issue for others when using k8.
I don't think you're being fair with your comparisons to DO when it comes to price. The f1 micro $5/month Google servers are SO much worse.
Those "always free" f1 micros have 600mb of RAM, do not have SSDs and according to some tests[0] you only get 20% of 1 vCPU. There's also some pretty severe write limitations.
That's a really big difference vs DO's 1gb of RAM, SSD and 100% of 1 vCPU. The performance difference alone with SSD vs no-SSD is massive, especially on a database and while most web apps aren't CPU bound, 20% of 1 vCPU is bound to run into problems beyond a simple hello world that you set up in your example.
Also what happens if your app doesn't even fit in that 600mb? On DO you could get a 600mb app + 100mb OS + 150mb DB + whatever else running on that $5/month server since it has 1gb of ram.
In a real world application I wouldn't be surprised if that cluster performs way worse than a single DO server in the $5/month range. In some cases your app wouldn't even be able to run.
This is part of the reason why I use a similar setup as OP, but with a single n1-standard-1 instance (1 vCPU/3.75 GB RAM) instead ($7.30/mo w/ preemptible). The 3-node requirement of GKE is not a hard requirement apparently.
Of course, that's mostly for running Jobs; a single node + preemptible with a production app is not a great idea.
I‘m running a full-featured three node cluster for less than $10/month on Hetzner Cloud. This kind of setup requires a little more effort, but it can be fully automated and you learn a couple of things about Kubernetes on the way. I published and maintain this guide[1], including provisioning using Terraform and a bunch of example manifests to get you started.
One thing you might want to look at based on a quick scan over your guide is that, at the moment ,it looks like you're running your etcd cluster with no authentication.
Any attacker who was able to get on to the pod network (e.g. if there's a vulnerability in one of the apps running on the cluster) could hit the etcd endpoint and dump the contents of the database, which generally includes sensitive information like k8s secrets.
Although this is a valid question, it doesn't apply here. See, I try to teach myself something new every year, whether it's a new programming language, platform or whatever else.
It was curiosity that turned my non-existent infrastructure into having a cheap, small-scale Kubernetes cluster ready for my shenanigans.
My tiny cluster gives me great freedom. Setup is fully automated. When a new k8s release comes around, I simply tear down my old cluster and ramp up a new one in mere minutes. If I want to deploy a service or application, all I need to do is to write a couple of manifests and roll all the moving parts out. Hassle-free - no SSH, no additional costs, no accounts, no 3rd-party services, persistent storage and TLS certificates out of the box.
Currently, a Unifi controller is the only long-term resident on my cluster; other services and projects come and go. Besides that, the following agents are deployed as Kubernetes pods: Datadog (free tier) for basic monitoring & alerting and Papertrail ($5-10/month) which acts as a centralized log sink for all my hosts, NAS and networking gear.
nice guide, the only problem i see with hetzner cloud is that they do not allow to have a operating system like coreos or even the coreos (Container Linux) fork flatcar or even RancherOS.
>> You don't have to learn systemd; you don't have to know what runlevels are or whether it was groupadd or addgroup; you don't have to format a disk, or learn how to use ps, or, God help you, vim
That's a quote from the article. The thought that someone with that mindset is responsible for anything more than a network-connected waffle iron is terrifying. This article advocates for willful ignorance of Unix/Linux, because you don't need any of those things if you know k8s.
That said, k8s is nice for some things. I admin a large deployment at work, and it's relatively painless for developers to be productive with. One of the reasons for that stability isn't k8s itself though - it's the legion of seasoned Unix/Linux pros working alongside me. The reason "it just works" is because under the covers, there are a whole lot of us doing all the boring stuff to hundreds of physical hosts.
I'm sympathetic to the author's argument, UNIX administration is a full time job on top of whatever system you're using to do deployments. You leverage your ops team at work; is ops-as-a-service really all that different?
If one of my devs wants to deploy a rails app I don't expect them to suddenly be a sysadmin or network engineer or infosec expert. I expect their application to be well written but the rest of the stack is up to ops.
True, deploying stuff in an actual Unix environment while ignoring need of these tools is the perfect recipe for unstable setups.
Although I guess for personal projects it could actually work as long as the load is low enough and when there is no data involved that needs to be protected. ;)
This article, and a lot of others, seem to be using an overloaded meaning of "personal".
All the arguments stem from the assumption that personal projects have the same uptime, scaling, and deployment requirements as some large commercial site. Some personal projects might but I'd argue most of those are really commercial in the sense that you're developing them for a portfolio or the like. They're not personal.
It makes sense to use business tools like kubernetes when you're making a demo business site. It doesn't make it a good platform for personal projects.
The best platform for those is hosting from home on a single computer, for free, with total control. ie, personal.
I disagree. Even for personal projects I want to be able to access them from my phone while out and about.
Something running on my home computer (which is turned off when I'm not there!) doesn't help. Nevermind that my home internet isn't fast or reliable.
If you include running my own blog/mail/XMPP/matrix servers in "personal projects", then I would love to have perfect uptime.
Additionally, being able to reuse high quality helm charts here just makes things simple instead of having one set of instructions for "personal" use and one set of instructions for everyone else.
Is your ISP and service really that bad? If it is I think you have bigger problems you should address before working on personal projects.
I've been running my personal website from home for almost 2 decades on Comcast and I've never had a problem. I can always access my sites when I'm not home. And additionally I can access all my other media. Even when I only had a megabit or so of upload it was more than enough. Downtimes have been very rare and usually only for a couple hours once per year in the middle of local night.
That said, of course you can't run a mailserver from home. For that, because of residential IP blacklists, I run from a rented VPS. In fact I run webservers on them too, but only for hosting large files. It's easier to just cp whatever ~/www than always have to upload things.
Why would you want to be able to access your personal projects from your phone? What context requires this over waiting and solving the problem at home?
What about work life balance? (yes, it's a personal project, but still)
Often personal projects are for learning new technologies and applying them in a meaningful way. E.g. many people want to run their personal projects on Kubernetes to play around with Kubernetes and understand how apps can be operated with it.
Kubernetes is a complicated, complex beast. Even on managed K8s on a cloud provider, you still need to learn a lot of stuff that has absolutely nothing to do with what you intend to accomplish (unless your personal project is learning to use K8s.
Most of my personal projects get deployed on Google's App Engine. It's easy, simple and unbelievably cheap.
There was a fantastic presentation at the last IO about riding as much as you can on free tiers:
Sure if you put enough effort in you can make everything usable and cheap. But why put in the effort into that instead of investing it into building a system that naturally fufills such requirements?
Kubernetes was never intended as a system for hobbyists, but for bringing more users onto google cloud. At that it is incredibly successful. But in exchange a lot of other stuff it does really badly.
It is much better to start with a single-node system in mind that expects getting more nodes at some point and develop from there. This will bring in a lot of hobbyists and really carves out how to solve the developer story. Then this will grow automatically with the size of popular projects building on it.
That said, of course it's nice if people invest energy into finding other usecases for k8s as well. But it shouldn't be the community's main effort.
Readit News
Having used it, there's a sizeable amount of further work needed which the article doesn't mention (e.g. learning how to use the pretty confusing google interface, finding the right logs and using their tools). So the overhead is really huge.
Furthermore, the whole system is slow. Want to run a SQL query against your postgres? You need to use a google cloud command that changes the firewall and ssh's you in on the machine... and this takes a couple of minutes, just enough to make me desist unless I _really_ need to run that query. Abysmal.
Finally, and this is a pet peeve against many advocacy blog posts, they just show you the happy path! Sure, _in the best of cases_ you just edit a file. In a more realistic case, you'll be stuck with a remote management system which is incredibly rich but also a really steep learning curve. Your setup is not performant? Good luck. Need to tweak or fine tune? Again, best of luck.
We've tried to adopt k8s 3-4 times at work and every single time productivity dropped significantly without having significant benefits over normal provisioning of machines. {Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!}
...which in turn is usually significantly slower than building your own home server (but that's another story!)
I know a bio scientist who spent two months working on containers and Docker and what not, for what are essentially independent shell scripts that are best run as batch jobs. I spoke with him in length, and he realized at the end of the day that what he really needed was better understanding of standard *nix processes and not containers...
- automated ssl
- centralized logging
- super easy scaling up and down (going from 1 instance to 2 instance is a headache manually)
- centralized auth (have a service which doesnt have built in auth?)
- super easy recovery (containers recreated/volumes attached automatically/apps just work unless you have db type of application which shouldn't be just restarted, which is rare)
- Smooth CI/CD (gogs/jenkins/private docker image repository)
As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in.
> I know a bio scientist who spent
Super obscure example. Kubernetes is definitely not for non-tech people. And I didn't pick k8s overnight. Spent a year doing playful POCs before deploying in a production environment.
If the only thing thats stopping you from using k8s is the learning curve, I suggest you to go ahead and learn it. Its a huge boon.
I mean, sure, kubernetes _is_ complex. But it's complex in the way that using Rails is more complex than pure Ruby. It's fine when you're playing, or you have a weird subset of problems where you don't care if someone runs or not, but as soon as you deal with the problems of HA, DR, live deployments, secret management, sharing servers, working in a team, etc then if you're solving these issues with ansible and terraform, you're probably just inventing something as complex, and worse, bespoke to your problem.
At the end of the day, once you've learned to use kube, it's probably no worse than any other configuration as code based system, and the question is which of us should be learning it and when, and which of us should just be using higher levels of abstractions like gitlabs autodevops.
Now, indeed, if you're just hacking and you have no need for a CI and CD pipeline, or reproducibility, etc, then sure, it's probably not the time to learn a hosting framework, but once you do know it, you're free to make an informed opinion on your problem.
Personally, I sell kube a lot, but I tend to work for orgs that have several development teams and several apps, and bringing them to a platform seems like a good idea. The question is, should I also put a startup on GKE, or should I do something bespoke where they can't hire another contractor to take over as easily? Personally, I'd go GKE.
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Over the years you easily end up with a pile of these projects, which want to keep running mostly for fun and therefore want to minimize the associated fixed costs. Using containers may help in keeping the herd manageable so that you can, for example, move everything to some other hosting provider.
Interestingly, while the author compares k8s to an SQL database, they actually do not deploy a DB. It's all fun an giggles until you deploy a DB.
Or, as worst, the shortcut to understanding the underlying tech. But, isn't that the point of abstraction?
and the people who, when they show up at their Google interview, nail the part about scaling. Not that Google ever asked me how to scale a project, but it doesn't hurt learning things just for the sake of learning.
So, what happened almost every time is that the business unknowingly pivoted into innovating on PaaS, because Docker/Kubernetes was only the first step. After that, you had to figure out the best approach to CI, how to set up your deploy pipelines across multiple environments, how to replicate what you were used to seeing with Heroku, etc.
And of course the cost balloons at that stage because your cheap 3 node setup ends up exposing half a dozen load balancers that each cost money, so you start to think about hosting Traefik or nginx-ingress-controller so you can manage load balancing yourself and save some money, and because it's so easy to add more services once you know how, you start wanting to self-host all the things.
Meanwhile your startup hasn't even launched to the public yet and the sunk cost fallacy has fully embedded itself in the team mentality: they've just put months of time and effort into setting up the perfect K8S/Docker architecture and workflow, that now requires actual devops experience to maintain, and you can't push back on it because it's all about scaling when things go live, and self-hosting, and how convenient it all is.
Except, you know, that's 3-6 months of runway down the drain because leadership didn't shift the focus back to an MVP and let the fantasy continue. And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for.
Exactly. I was following along with the article until he mentioned that startups are doing this. Using k8s at a startup (especially a startup building an MVP) is just like using microservices. It is a disservice to the business by technologists.
Building the monolith on heroku will get you to market faster.
Yagni. (Until you do, and thrn by all means pay off that technical debt.)
https://captainduckduck.com
https://github.com/githubsaturn/captainduckduck
It made it so much easier to deploy new changes (just `captainduckduck deploy` and done). We also use minio.io (open-source S3) and are extremely productive with those tools.
It's perfect for web agencies. It's not sophisticated enough for k8s' use case, but it's extremely easy to use (you can just add nodes and it will restart containers automatically [1]).
[1]: https://captainduckduck.com/docs/app-scaling-and-cluster.htm...
D) Disable Firewall Some server providers have strict firewall settings. To disable firewall on Ubuntu: ufw disable
WTF?
We need to go back and understand the problem we’re trying to solve, and for 99% of smallish companies and projects this is not container orchestration or hardware resource abstraction.
And most often all those app or two are doing is converting JSON to SQL and SQL to JSON actually. And PostgREST would be pretty much enough for that.
Admittedly, operation via Rancher comes with its own complexities: setting up security groups and IAM roles. These are documented of course, but as you say: minimum effort remains non-trivial.
I'll grant that there can be headaches that I don't really see anymore because I was deeply immersed in it for a while. But running a linux server by itself was hard till I learned that too.
Performance is a subtler point, not sure I follow what you're trying to say there. Container overhead? At least in my experience gke seemed responsive, and deployments settled pretty quick.
Tweaking is usually a strong point of k8s. Just reapply... so I'm probably misunderstanding what you were getting at here.
Kubernetes needs to improve their accessibility to newcomers but the happy path isn't unheard, for simple stateless apps it mostly just works. And when it doesn't I can usually kubectl log and kubectl delete pod my way to a solution and there are docs and stack overflow to help.
Not perfect but I was surprised it worked as well as it did.
Best solution is Cloud SQL Proxy: https://cloud.google.com/sql/docs/postgres/sql-proxy
We thought about k8s a couple of times but it always looked too over complicated and no gain compared to our use of DCOS
That said, once you learn all of these k8s concepts and mechanisms, and probably script nearly all of them, they all become like breathing and don't really take much time.
I frankly now am at the point where it is faster to spin up a small project on k8s than it would be to do so with the cloud providers PaaS platform.
Properly hosting custom apps on the web in a resilient way isn't easy. This being the case, it seems an error to interpret the thoroughness of the article as a signal that the proposed solution or situation is unreasonably complex.
I'll take detailed and exhaustive over sparse and hand-wavy any day!
Kubernetes certainly has a learning curve, just like most worthwhile things in life.
That being said, there are certainly existing solutions for doing that, for cheap or even free tiers with less effort - Heroku, Google App Engine, etc.
It can be rough to manage Kubernetes when you just want to focus on your app. I was having that same problem when building side projects and focused on building a platform to deploy dockerfiles on a managed Kubernetes instance.
https://usekube.co
> Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!
Or it could mean you're not the right people for it?
Do you have a dedicated individual or team that represents infrastructure engineering? Are you a team of developers trying to play the Ops game? More often than not people/companies that struggle to get K8s going or use it are a team of developers who's expertise is code and not infrastructure/automation.
K8s is hard when done from scratch (there's a book and a video course about that by THE K8s guy.) It's not hard when you use Kops, Tectonic (discontinued, I believe), EKS, or one of the other managed services. Implementing a YAML file and launching it isn't hard. Also, you don't have to understand it inside out; you only have to understand how-to use it. Can you tell me how every component in your microwave works or do you just get on with cooking your food?
And to say you're not the use case for it - that's highly unlikely. I mean, it's possible, but it's unlikely. Your work loads and software solutions would have to be highly unique and or specialised because hey, it's not like Google's setup is simple, right?
Great idea! We could call that team 'Operations' and they would do all that stuff so developers don't have to!
I'm skipping job postings which require k8s. It just tells me they're not competent, falling for the hype. For AWS, ECS is way simpler and is a fully featured orchestration tool. I had an interview today, where I tried to educate why simple is better, and why they don't need k8s. No luck :)
If you're doing individual development, however, it's quite likely that you're also doing development of the database, and need to do ad-hoc data analysis before you include those queries in that app.
"Then don't do it. It only shows you're an idiot anyway."
Not a good answer.
1. Something that makes it easy to run the application(s) you're working on locally... 2. ...but have it interact with services in a local kubernetes cluster easily. 3. Something that makes your entire development environment effectively immutable, read-from-files the same way as the immutable infrastructure we're talking so much about. 4. Something that gives you a good start to running in production, so the first day of bringing your service up isn't a mess of trying to find/remember what environment variables/config options to set. 5. Something that's one tool, instead of a mixture of `minikube`, `kubectl`, `docker`, `draft`, and `telepresence`. 6. Something that's opinionated, and follows "convention over configuration", with conventions that make for the best future designs.
Basically, we need something with a Rails-like impact but in the Kubernetes ecosystem.
Swarm is such a pleasant and easy alternative that comes out of the box with Docker, does 90% of what Kubernetes does, and does 100% of what anyone with less than 50 servers needs. The learning curve is also an order of magnitude easier.
As for an app deployment tool I've had good success with Helm. It allows you to define all the services, app, variables, etc. that your app needs. Though the templates do seem to be a bit convoluted and hard to read at times if you aren't careful.
2. Haven't tried this, but I'm aware of it - does this address your point here ? https://www.telepresence.io/
3. Something like this? https://github.com/rycus86/docker-pycharm I agree that this could be very cool if further developed.
I agree that we'll start to see deeper integration across the development environment though, as more dev-hours are spent on building up the toolchain.
# Local development / testing
I use "minikube" for developing and testing each service locally. I use a micro service architecture in which each service can be tested in isolation. If all tests pass, I create a Helm Chart with a new version for the service and push it to a private Helm Repo. This allows for fast dev/test cycles.
These are the tasks that I run from a "build" script:
* install: Install the service in your minikube cluster
* delete: Delete the service from your minikube cluster
* build: Build all artifacts, docker images, helm charts, etc.
* test: Restart pods in minikube cluster and run tests.
* deploy: Push Helm Chart / Docker images to private registry.
This fits in a 200 LOC Python script. The script relies on a library though, which contains most of the code that does the heavy lifting. I use that lib for for all micro-services which I deploy to k8s.
# Testing in a dev cluster
If local testing succeeds, I proceed testing the service in a dev cluster. The dev cluster is a (temporary) clone of the production cluster, running services with a domain-prefix (e.g. dev123.foo.com, dev-peter.foo.com). You can clone data from the production cluster via volume snapshots if you need. If you have multiple people in your org, each person could spawn their own dev clusters e.g. dev-peter.foo.com, dev-sarah.foo.com.
I install the new version of the micro-service in the dev-cluster via `helm install` and start testing.
These are the steps that need automation for cloning the prod cluster:
* Register nodes and spawn clean k8s cluster.
* Create prefixed subdomains and link them to the k8s master.
* Create new storage volumes or clone those from the production cluster or somewhere else.
* Update the domains and the volume IDs and run all cluster configs.
I haven't automated all of these steps, since I don't need to spawn new dev clusters too often. It takes about 20 minutes to clone an entire cluster, including 10 minutes of waiting for the nodes to come up. I'm going to automate most of this soon.
# Deploy in prod cluster
If the above tests pass I run `helm upgrade` for the service in the production cluster.
This works really well.
I was really hyped about Azure Dev Spaces (https://docs.microsoft.com/en-us/azure/dev-spaces/), but it's Azure only, even though it sounds like something that should work on any k8 cluster..
One vagrant VM allows you to make assumptions about what software is installed on a dev desktop which is especially useful if you're writing documentation. Getting into the Vagrant VM from various OSes is the main problem but once you're in... Then you can install container software and teach others how to work in docker given one assumed OS.
Since Vagrant and containers encourage some form of infrastructure/documentation as code, then more experienced developers can read through what goes into the Vagrant set up and replicate on their native OS if they prefer. As someone writing documentation, I can forget about those more experienced devs and just write to one OS.
Once you have documentation that you can guarantee works when using container software, much of the headache can be swept under the rug for more junior engineers which I think is the krux of the issue for others when using k8.
I'd very much like a Ruby/Python/Rust of cloud infrastructure to come along and clean up this mess.
Deleted Comment
Those "always free" f1 micros have 600mb of RAM, do not have SSDs and according to some tests[0] you only get 20% of 1 vCPU. There's also some pretty severe write limitations.
That's a really big difference vs DO's 1gb of RAM, SSD and 100% of 1 vCPU. The performance difference alone with SSD vs no-SSD is massive, especially on a database and while most web apps aren't CPU bound, 20% of 1 vCPU is bound to run into problems beyond a simple hello world that you set up in your example.
Also what happens if your app doesn't even fit in that 600mb? On DO you could get a 600mb app + 100mb OS + 150mb DB + whatever else running on that $5/month server since it has 1gb of ram.
In a real world application I wouldn't be surprised if that cluster performs way worse than a single DO server in the $5/month range. In some cases your app wouldn't even be able to run.
[0]: https://www.opsdash.com/blog/google-cloud-f1-micro.html
Of course, that's mostly for running Jobs; a single node + preemptible with a production app is not a great idea.
I‘m running a full-featured three node cluster for less than $10/month on Hetzner Cloud. This kind of setup requires a little more effort, but it can be fully automated and you learn a couple of things about Kubernetes on the way. I published and maintain this guide[1], including provisioning using Terraform and a bunch of example manifests to get you started.
[1] https://github.com/hobby-kube/guide
Any attacker who was able to get on to the pod network (e.g. if there's a vulnerability in one of the apps running on the cluster) could hit the etcd endpoint and dump the contents of the database, which generally includes sensitive information like k8s secrets.
Edit: I think this was you.
It was curiosity that turned my non-existent infrastructure into having a cheap, small-scale Kubernetes cluster ready for my shenanigans.
My tiny cluster gives me great freedom. Setup is fully automated. When a new k8s release comes around, I simply tear down my old cluster and ramp up a new one in mere minutes. If I want to deploy a service or application, all I need to do is to write a couple of manifests and roll all the moving parts out. Hassle-free - no SSH, no additional costs, no accounts, no 3rd-party services, persistent storage and TLS certificates out of the box.
Currently, a Unifi controller is the only long-term resident on my cluster; other services and projects come and go. Besides that, the following agents are deployed as Kubernetes pods: Datadog (free tier) for basic monitoring & alerting and Papertrail ($5-10/month) which acts as a centralized log sink for all my hosts, NAS and networking gear.
That's a quote from the article. The thought that someone with that mindset is responsible for anything more than a network-connected waffle iron is terrifying. This article advocates for willful ignorance of Unix/Linux, because you don't need any of those things if you know k8s.
That said, k8s is nice for some things. I admin a large deployment at work, and it's relatively painless for developers to be productive with. One of the reasons for that stability isn't k8s itself though - it's the legion of seasoned Unix/Linux pros working alongside me. The reason "it just works" is because under the covers, there are a whole lot of us doing all the boring stuff to hundreds of physical hosts.
If one of my devs wants to deploy a rails app I don't expect them to suddenly be a sysadmin or network engineer or infosec expert. I expect their application to be well written but the rest of the stack is up to ops.
Although I guess for personal projects it could actually work as long as the load is low enough and when there is no data involved that needs to be protected. ;)
All the arguments stem from the assumption that personal projects have the same uptime, scaling, and deployment requirements as some large commercial site. Some personal projects might but I'd argue most of those are really commercial in the sense that you're developing them for a portfolio or the like. They're not personal.
It makes sense to use business tools like kubernetes when you're making a demo business site. It doesn't make it a good platform for personal projects.
The best platform for those is hosting from home on a single computer, for free, with total control. ie, personal.
If you include running my own blog/mail/XMPP/matrix servers in "personal projects", then I would love to have perfect uptime.
Additionally, being able to reuse high quality helm charts here just makes things simple instead of having one set of instructions for "personal" use and one set of instructions for everyone else.
I've been running my personal website from home for almost 2 decades on Comcast and I've never had a problem. I can always access my sites when I'm not home. And additionally I can access all my other media. Even when I only had a megabit or so of upload it was more than enough. Downtimes have been very rare and usually only for a couple hours once per year in the middle of local night.
That said, of course you can't run a mailserver from home. For that, because of residential IP blacklists, I run from a rented VPS. In fact I run webservers on them too, but only for hosting large files. It's easier to just cp whatever ~/www than always have to upload things.
What about work life balance? (yes, it's a personal project, but still)
Or just get a cheap DO box. You can run quite a lot one, and have it always accessible, on the internet, etc...
Kubernetes is a complicated, complex beast. Even on managed K8s on a cloud provider, you still need to learn a lot of stuff that has absolutely nothing to do with what you intend to accomplish (unless your personal project is learning to use K8s.
Most of my personal projects get deployed on Google's App Engine. It's easy, simple and unbelievably cheap.
There was a fantastic presentation at the last IO about riding as much as you can on free tiers:
https://www.youtube.com/watch?v=N2OG1w6bGFo
Kubernetes was never intended as a system for hobbyists, but for bringing more users onto google cloud. At that it is incredibly successful. But in exchange a lot of other stuff it does really badly.
It is much better to start with a single-node system in mind that expects getting more nodes at some point and develop from there. This will bring in a lot of hobbyists and really carves out how to solve the developer story. Then this will grow automatically with the size of popular projects building on it.
That said, of course it's nice if people invest energy into finding other usecases for k8s as well. But it shouldn't be the community's main effort.