I've definitely experienced the problems of using Docker Compose for local dev. Oh, that file changed? Control-C to kill the whole cluster. Wait until the DB flushes. Start it again.
And if you just want to restart one service? Well you have to run 4 commands[0] in order to properly do it...
And what the author was saying about needing to "template" the YAML files is also true. In the past, when shipping an Open Source library for others to use, we had to wrap Docker Compose in a CLI that generates the YAML file and then invokes Docker Compose for you. (See it here[1][2][3])
It sucks and this library definitely solves a real problem!
These are exactly the kinds of issues that drove me to work on miniboss. My top grievance has been the "waiting for service available" issue, which is generally solved by wrapping the dependent code in a `wait-for-it.sh` bash script. A rather hacky solution for a rather simple problem.
Hahaha yes! I had to add all of that to our code as well. I also had to write a "wait-for-file.sh" script for a mechanism to pass config values. (We use LocalStack to host an AWS env locally and the S3 bucket name is random.)
In Docker Compose v2 they had a more sophisticated "depends on" model with health checks, but they removed it in v3. The docs say something about Docker Swarm so I'm guessing they removed that in order to try to make v3 syntax work for clusters too.
The real tragedy here is that Kubernetes won, but we're still stuck with Docker Compose trying to be the same thing.
I hope your tool can help change this annoying status quo!
Not 4, but for me at least it's often 3. docker-compose has a bad habit of re-using the existing container, i.e. `docker-compose restart <service>` won't stop the existing service container and create a new one, it restarts the current container. Often the reason I'm restarting the service is because I need a clean, new container to pick up some env change or for some debugging or something, so habitually I restart containers with `docker-compose stop <service> && docker-compose rm -f <service> && docker-compose up -d <service>`.
I mostly like docker-compose just fine, but this is an annoyance & when I first started using docker-compose some years back I definitely lost more time than I would have liked trying to figure out why some change I'd just made didn't seem to have applied. I feel like destroying/creating containers is pretty cheap (that's one of the great things about containers!), so I'm not sure why docker-compose behaves this way. I presume it's for historical reasons and they don't feel comfortable changing it because of backwards compatibility.
It's potentially possible. I'd have to look at it again to verify.
What we were struggling with were "conditional services" like running Integration Tests. We only wanted to add that container to the cluster when we wanted to run the tests.
Same with the "demo" mode we added. That spins up an Nginx container with an example front-end app.
It's theoretically possible to define all of the containers in one big Docker Compose config and then turn them on/off with env variables per environment, but it's just hellish to debug. In fact, I suspect that's what the engineer that wrote those started with before getting frustrated with spaghetti.
Sometimes there is just no winning with the computers lol
Edit: Answering the "it isn't 4 commands" bit -- if you just restart a container it won't pick up changes to the Docker Compose config. You have to remove it an add it back to the cluster.
While it's probably fine for most of the use cases you're predicting, if someone who isn't quite familiar with Python's gotchas decides to subclass, or modify one of the items at runtime, it will be a tricky thing to troubleshoot. Even though that is a rite of passage for Python developers, it's probably best to not have examples that could set them up for weird bugs.
edit:
I was talking about mutable types at the top level of a class, and somehow left that out.
Good point. I was thinking of adding `docker-compose` compatibility, so that you can use a `docker-compose.yml` file to define services, and then add event hooks etc. with decorators, or something similar. You would then also have access to the attributes of the services.
I meant to say that having mutable types defined at the top level of a class was a problem, I must have backspaced over the most important part while I was figuring out to say it.
Classes in and of themselves are great, but having lists and dicts defined at the top level can cause problems.
I actually dug around in the source code of docker-compose to figure out how to properly work with networks. However the fundamental ideas of the two projects are different (yaml -> containers vs. Python code -> containers), so starting off from docker-compose just wasn't right. It would have simply complicated things.
Have you thought about taking an approach similar to the AWS CDK with CloudFormation? Aka the whole "infrastructure as code" movement?
I'm not sure how that would work with the whole "lifecycle hooks" you've mentioned a few times, but maybe it would be easier than trying to compete with Docker Compose yourself by simply wrapping it.
I posted in another comment here too but we wrote a bunch of code to deal with programmatic generation of Docker Compose files, and it was really sweet to use! I've honestly thought about making that code[0] a stand-alone library because of how valuable it was.
I have no idea, as I'm a bit lost when it comes to Kubernetes tooling. When I wrote a Kubernetes tutorial (https://okigiveup.net/tutorials/a-tutorial-introduction-to-k...) nearly four years ago, you had kubectl and YAML files, and that was it. These days there is a whole hierarchy of tools, which probably cover the same ground with miniboss and much more. So yes, it could be adapted, but I'm pretty sure it wouldn't be necessary.
I have no idea what kompose convert does, honestly. Principally, one could print out any format from miniboss, but it depends on how much you want to trust a tool written for local deployments with kubernetes stuff.
Checkout kapitan, with its Kadet input type, you can generate YAML with a semi-opinionated Python based DSL: https://kapitan.dev/compile/#kadet
I am the author and have used it at a FAANG previously and now more recently at my startup to manage large, complex kubernetes and terraform managed infrastructure.
Check out Pulumi, it's a similar idea of python (or other languages) to generate container orchestration config for Kubernetes or other systems: https://github.com/pulumi/pulumi
I would say it's more similar to Pulumi, as the main aim is to avoid markup (or a simplified programming language, like HCL in the case of terraform). The other two are correct though: Only for Docker and in Python.
Readit News
And if you just want to restart one service? Well you have to run 4 commands[0] in order to properly do it...
And what the author was saying about needing to "template" the YAML files is also true. In the past, when shipping an Open Source library for others to use, we had to wrap Docker Compose in a CLI that generates the YAML file and then invokes Docker Compose for you. (See it here[1][2][3])
It sucks and this library definitely solves a real problem!
0: https://stackoverflow.com/questions/31466428/how-to-restart-...
1: The code: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa...
2: The package it lives in: https://github.com/lunasec-io/lunasec/tree/master/js/sdks/pa...
3: The docs for the wrapped CLI: https://www.lunasec.io/docs/pages/overview/demo-app/overview
In Docker Compose v2 they had a more sophisticated "depends on" model with health checks, but they removed it in v3. The docs say something about Docker Swarm so I'm guessing they removed that in order to try to make v3 syntax work for clusters too.
The real tragedy here is that Kubernetes won, but we're still stuck with Docker Compose trying to be the same thing.
I hope your tool can help change this annoying status quo!
Care to elaborate? The SO answers seem to indicate it can be done in a single command, and my personal experience confirms that.
Are you perhaps also including pulling new images, rebuilding local Dockerfiles, etc etc as part of the “restart service” process?
Restarting one service isn't 4 commands.
I mostly like docker-compose just fine, but this is an annoyance & when I first started using docker-compose some years back I definitely lost more time than I would have liked trying to figure out why some change I'd just made didn't seem to have applied. I feel like destroying/creating containers is pretty cheap (that's one of the great things about containers!), so I'm not sure why docker-compose behaves this way. I presume it's for historical reasons and they don't feel comfortable changing it because of backwards compatibility.
What we were struggling with were "conditional services" like running Integration Tests. We only wanted to add that container to the cluster when we wanted to run the tests.
Same with the "demo" mode we added. That spins up an Nginx container with an example front-end app.
It's theoretically possible to define all of the containers in one big Docker Compose config and then turn them on/off with env variables per environment, but it's just hellish to debug. In fact, I suspect that's what the engineer that wrote those started with before getting frustrated with spaghetti.
Sometimes there is just no winning with the computers lol
Edit: Answering the "it isn't 4 commands" bit -- if you just restart a container it won't pick up changes to the Docker Compose config. You have to remove it an add it back to the cluster.
While it's probably fine for most of the use cases you're predicting, if someone who isn't quite familiar with Python's gotchas decides to subclass, or modify one of the items at runtime, it will be a tricky thing to troubleshoot. Even though that is a rite of passage for Python developers, it's probably best to not have examples that could set them up for weird bugs.
edit: I was talking about mutable types at the top level of a class, and somehow left that out.
Classes in and of themselves are great, but having lists and dicts defined at the top level can cause problems.
You could keep the YAML format and extend it, and still write compose files as pure Python.
I'm not sure how that would work with the whole "lifecycle hooks" you've mentioned a few times, but maybe it would be easier than trying to compete with Docker Compose yourself by simply wrapping it.
I posted in another comment here too but we wrote a bunch of code to deal with programmatic generation of Docker Compose files, and it was really sweet to use! I've honestly thought about making that code[0] a stand-alone library because of how valuable it was.
0: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa...
How tied is it to the Docker daemon?
Instead of Docker, could one use podman, which doesn't need root.
It's docker-compose compatibility is tied up in essentially re-implementing/emulating that, which partially defeats the purpose.
My main need for the daemon component is https://github.com/nginx-proxy/acme-companion which listens to docker events to know when a new service is available to proxy.
I've managed to never touch a Kubernetes yaml thanks to it.
I am the author and have used it at a FAANG previously and now more recently at my startup to manage large, complex kubernetes and terraform managed infrastructure.