I don't think I agree with this. Docker is an amazing tool, I've used it for everything I've done in the last 7 years, but this is not how I'd approach it.
1. I think the idea of local-equal-to-prod is noble, and getting them as close as possible should be the goal, but is not possible. In the example, they're using a dockerized postgres, prod is probably a managed DB service. They're using docker compose, prod is likely ECS/K8S/DO/some other service that uses the image (with more complicated service definitions). Local is probably some VM linux kernel, prod is some other kernel. Your local dev is using mounted code, prod is probably baked in code. Maybe local is ARM64, and prod is AMD64.
I say this not because I want to take away from the idea of matching dev and prod as much as possible, but to highlight they're inherently going to be very different. So deploying your code with linters, or in debug mode, and getting slower container start times at best, worse production performance at worse - just to pretend envs which are wildly different aren't different seems silly. Moreover if you test in CI, you're much more likely to get to a prod-like infra than a laptop.
2. Cost will also prohibit this. Do you have your APM service running on every dev node, are you paying for that for all the developer machines for no benefit so things are the same. If you're integrating with salesforce, do you pay for a sandbox for every dev so things are the same. Again, keeping things as similar as possible should be a critical goal, but their are cost realities that again make that impossible to be perfect.
3. In my experience if you actually want to achieve this, you need a remote dev setup. Have your code deployed in K8S / ECS / whatever with remote dev tooling in place. That way your DNS discovery is the same, kernels are the same, etc. Sometimes this is worth it, sometimes it isn't.
I don't want to be negative, but if one of my engineers came to me saying they wanted to deploy images built from their machine, with all the dev niceties enabled, to go to prod, rather than proper CI/CD of prod optimized images, I'd have a hard time being sold on that.
After going through a bunch of evolutions using Docker as co-founder/engineer #1 at a startup to > 100 engineers, hard agree on this take.
One other reason to not overbloat your images (besides physical storage cost and perf) is security considerations. If you find yourself in a place where you need to meet enterprise security standards, keeping more dependencies in your image and linked to your app code widens your risk vector for vulnerabilities.
> I don't want to be negative, but if one of my engineers came to me saying they wanted to deploy images built from their machine, with all the dev niceties enabled, to go to prod, rather than proper CI/CD of prod optimized images, I'd have a hard time being sold on that.
Whereas this was true (for a long time), making an argument that because remote and local are so “inherently” different that one shouldn’t strive for this parity is silly, especially considering the differences outlined are pretty easily solvable by k8s to local parity.
Whereas it’s still clunky to dev with tools like skaffold and minikube, I strongly believe they are the future. We have essentially eliminated deployment bugs using skaffold for local dev and deployment. Everything is caught locally, on a dev machine or in CI, as it should be.
"widely different" seems like a stretch e.g. ECS is pretty directly translatable to docker compose, and if you do cross-platform builds with buildx then I don't see why doing the building locally or on a cloud service matters much.
This demonstrates the most pernicious thing about Docker: it is now easier than ever for someone to design a Rube Goldberg machine and then neatly sweep it under the rug.
When you see the dev environment setup described, the knee jerk reaction should be to simplify it, not to automate the running of 30 disparate commands. Then you can much more easily run it in production, instead of boxing up the mess and waiting until you actually have to debug it.
There’s really nothing complicated in the docker-compose listed. It’s simple, uses some very simple commands that everyone should know and sets environment variables at build.
We also have code reviews. They’re helpful with containers because sometimes people do silly things when they can. But that’s why we have code reviews in the first place.
I mean, having a web, worker, cache, db, and search as separate things doesn't seem that crazy?
While of course you can often get away with just a web with sqlite, pretty much any company of scale I've been at maintain those all as separate things, with separate search as the most optional.
What do you feel about the setup described seems overly complicated?
Including developer tooling in a Docker image is missing one of the really useful things about Docker: not needing all that stuff. By using a multi-stage build, you can do all the slow dev stuff at image build time, then only include the output in the image, and that includes things like building a library which wants a different set of conditions to build than your application wants to run.
It also adds an additional level of risk - if your image is compromised, but all that is running in it is your app, oh well. If it's compromised, and it's able to call out to other parts of your stack (yes, some of this is down to the specific deployment process), that's much worse.
That's a good idea. I usually have a 'myproject-debugtools' container that just operates on the same volume and maybe even shares a network with the 'myproject-prod' container. Just set it to `--restart no` and even when someone forgets to shut it down it'll be gone on reboot. That way all the non-prod stuff isn't even in the image at any point.
Or if that is too much work just have a 'myproject-dev' image/tag if you need to debug a live environment.
Docker is a deployment mechanism. This means publishing Docker images is a deployment activity not a development one.
I don't think software developers should publish Docker images at all [1]. This is a huge impedance mismatch with serious security implications. In particular, your Docker image needs a regular release cadence that is different from your software releases.
Including a Dockerfile is fine, they allow the person doing the deployment to customize/rebuild the image as needed (and help with development and testing too).
[1]: Though I'm not saying you can't be both a developer and sysadmin in your organization. Are you?
Agreed. Packaging is different than deployment. Devs should return to the art of packaging, such that their software can be then deployed into containers, VMs, micro VMs, whatever. That is what packaging allows, re-use.
This is the sort of behavior Nix encourages (disclaimer: I work at https://flox.dev , using Nix as our baseline tech). Docker as both a packaging and deployment format can carry a bit of weight, but can quickly get out of hand.
I love Docker (more so the idea of containers). Use it almost everywhere: self hosting services, at work everything is deployed as a docker container.
Except local development. Absolutely hate the "oh need to add a dependency, gotta rebuild everything" flow.
I do use it if the project I'm developing against needs a DB/redis/etc, but I don't think there's a chance I'm going back to using it for local development.
In fact, at work, the project where we do use docker in development actually causes the most headaches getting up and running.
I use a combination of CPU architectures, so the idea of running _exactly_ what's in production when developing is already out the window.
I hate container and Docker in ANY use-case where there is an alternative that is same, or even "a little bit" more involved.
I reserve Docker and Containers for that use case where I really would have headaches if it is no there, and have not still found such a case in all the works I've done.
One thing Docker helps with is reproducibility. If you write your images properly (not many people do) then you can have exact same conditions for every time you run tests. If you keep databases on the host machine, instead of containers, you will have to have some cleanup steps and automate somehow, that they are always run. Otherwise you risk shaky test results or even false positives/negatives. That might be fine, if the CI runs the tests reliably as well though.
Containers are great interface with other teams – ie black box their services, don't care how their things are running, just communicate which envs to use to make it work according to comms spec.
Docker for local development is only useful for running services like Postgresql and Redis, or doing hot reloads using something like vite or air. The development in a box paradigm is really difficult to maintain, much prefer direnv or nix.
Yup, at work we've been doing "dev machine bootstrap script installs version managers (nvm/mise/direnv/etc), projects use those" and have been experimenting with direnv + nix.
At least for python, I typically just add another RUN statement instead of changing a file used in a layer up the layer stack. That way the change is fast.
Then when I need to commit, I'll update the requirements file or whatever would cause all the layers to rebuild. And CI can rebuild the whole thing while I move to something else.
It is a bit of a pain, but the other benefits of containers are probably worth the trade off.
> What if running the linters was as easy as: $ docker compose exec web /scripts/run-linters
This seems to ignore the fact that I also run linters in my IDE to get immediate feedback as I’m writing code. As far as I know there’s no way to combine these two approaches. Currently I’m just careful to make sure my local ruff version matches the one used in CI.
It may be possible with VS Code
dev containers, but last time I looked at those I was turned off by the complexity.
This is all very good and true, but as usual the devil is found in the details. For instance, my company sells Docker images that depend on a very old and recently unmaintained binary. Over the years, I've found issues with that binary that make it very hard to be sure issues are completely reproducible from system to system (or, as the article suggests, from local to production). Sometimes, it's as simple as a newer base image updating a core dependency (e.g. Alpine updating musl), but other times it seems like nothing changes but the host machine, and diagnosing kernel-level issues - say, your local Mac OS' LinuxKit kernel versus your production Amazon Linux or Ubuntu, and don't forget x86 emulation! - make "test what you develop and deploy what you test" occasionally very daunting.
These are the sort of issues that Nix <https://nixos.org/> solves quite well. Pinning dependencies to specific versions so the only time dependencies change is when you explicitly do it - and the only packages present in your images are ones you specifically request, or dependencies of those packages. It also gives you local dev environments using the ~same dependencies by typing `nix develop`.
Once you get past the bear that is the language, it's a great tool.
I found setting up nix shells to be more time consuming than docker setups. Nixpkgs can require additional digging to find the correct dependencies that just work on other distributions. That being said, I’m a huge fan of NixOS, but I haven’t seen it as a replacement for docker for reproducible dev environments yet.
I'll grant that the kernel version+config shifting is a pain point, but I'd expect that containers help with the rest of it (userspace)? Yes, obviously changing the base image is a potential breaking change, but with containers you package up the ancient binary and the base image and any dependencies into a single unit, and then you can test that that whole unit works (including "did that last musl upgrade break the thing?"), and if it passes then you ship the whole image out to your users safe in the knowledge that the application will only be exposed to the libraries you tested it against and no newer versions.
Sounds like yall are doing a poor job building the container. It's one thing to rely on built in musl/glibc if it's modern software. However, if you are dragging technical debt, all those dependencies should be hard locked to the proper version.
What do people do for making these kinds of commands less verbose and easy to remember?
We've done things like use Makefile with the above behind `make lint`. However, chaining together shortcuts like "make format lint test" gets slow because "docker compose" for each one takes time to start up.
If you instead run the Makefile while you have a terminal open inside one of the Docker containers, that can be faster as you can skip the "docker compose" step, but then not every Makefile target will be runnable inside a Docker container (like a target to rebuild the Docker image), so you have to awkwardly jump between terminals that are inside/outside the Docker container for different tasks? Any tricks here?
Readit News
1. I think the idea of local-equal-to-prod is noble, and getting them as close as possible should be the goal, but is not possible. In the example, they're using a dockerized postgres, prod is probably a managed DB service. They're using docker compose, prod is likely ECS/K8S/DO/some other service that uses the image (with more complicated service definitions). Local is probably some VM linux kernel, prod is some other kernel. Your local dev is using mounted code, prod is probably baked in code. Maybe local is ARM64, and prod is AMD64.
I say this not because I want to take away from the idea of matching dev and prod as much as possible, but to highlight they're inherently going to be very different. So deploying your code with linters, or in debug mode, and getting slower container start times at best, worse production performance at worse - just to pretend envs which are wildly different aren't different seems silly. Moreover if you test in CI, you're much more likely to get to a prod-like infra than a laptop.
2. Cost will also prohibit this. Do you have your APM service running on every dev node, are you paying for that for all the developer machines for no benefit so things are the same. If you're integrating with salesforce, do you pay for a sandbox for every dev so things are the same. Again, keeping things as similar as possible should be a critical goal, but their are cost realities that again make that impossible to be perfect.
3. In my experience if you actually want to achieve this, you need a remote dev setup. Have your code deployed in K8S / ECS / whatever with remote dev tooling in place. That way your DNS discovery is the same, kernels are the same, etc. Sometimes this is worth it, sometimes it isn't.
I don't want to be negative, but if one of my engineers came to me saying they wanted to deploy images built from their machine, with all the dev niceties enabled, to go to prod, rather than proper CI/CD of prod optimized images, I'd have a hard time being sold on that.
One other reason to not overbloat your images (besides physical storage cost and perf) is security considerations. If you find yourself in a place where you need to meet enterprise security standards, keeping more dependencies in your image and linked to your app code widens your risk vector for vulnerabilities.
ditto, "worked on local" is a meme for a reason.
Whereas it’s still clunky to dev with tools like skaffold and minikube, I strongly believe they are the future. We have essentially eliminated deployment bugs using skaffold for local dev and deployment. Everything is caught locally, on a dev machine or in CI, as it should be.
When you see the dev environment setup described, the knee jerk reaction should be to simplify it, not to automate the running of 30 disparate commands. Then you can much more easily run it in production, instead of boxing up the mess and waiting until you actually have to debug it.
We also have code reviews. They’re helpful with containers because sometimes people do silly things when they can. But that’s why we have code reviews in the first place.
While of course you can often get away with just a web with sqlite, pretty much any company of scale I've been at maintain those all as separate things, with separate search as the most optional.
What do you feel about the setup described seems overly complicated?
It also adds an additional level of risk - if your image is compromised, but all that is running in it is your app, oh well. If it's compromised, and it's able to call out to other parts of your stack (yes, some of this is down to the specific deployment process), that's much worse.
Or if that is too much work just have a 'myproject-dev' image/tag if you need to debug a live environment.
I don't think software developers should publish Docker images at all [1]. This is a huge impedance mismatch with serious security implications. In particular, your Docker image needs a regular release cadence that is different from your software releases.
Including a Dockerfile is fine, they allow the person doing the deployment to customize/rebuild the image as needed (and help with development and testing too).
[1]: Though I'm not saying you can't be both a developer and sysadmin in your organization. Are you?
This is the sort of behavior Nix encourages (disclaimer: I work at https://flox.dev , using Nix as our baseline tech). Docker as both a packaging and deployment format can carry a bit of weight, but can quickly get out of hand.
Except local development. Absolutely hate the "oh need to add a dependency, gotta rebuild everything" flow.
I do use it if the project I'm developing against needs a DB/redis/etc, but I don't think there's a chance I'm going back to using it for local development.
In fact, at work, the project where we do use docker in development actually causes the most headaches getting up and running.
I use a combination of CPU architectures, so the idea of running _exactly_ what's in production when developing is already out the window.
I reserve Docker and Containers for that use case where I really would have headaches if it is no there, and have not still found such a case in all the works I've done.
Then when I need to commit, I'll update the requirements file or whatever would cause all the layers to rebuild. And CI can rebuild the whole thing while I move to something else.
It is a bit of a pain, but the other benefits of containers are probably worth the trade off.
This seems to ignore the fact that I also run linters in my IDE to get immediate feedback as I’m writing code. As far as I know there’s no way to combine these two approaches. Currently I’m just careful to make sure my local ruff version matches the one used in CI.
It may be possible with VS Code dev containers, but last time I looked at those I was turned off by the complexity.
Once you get past the bear that is the language, it's a great tool.
> $ docker compose exec web /scripts/run-linters
What do people do for making these kinds of commands less verbose and easy to remember?
We've done things like use Makefile with the above behind `make lint`. However, chaining together shortcuts like "make format lint test" gets slow because "docker compose" for each one takes time to start up.
If you instead run the Makefile while you have a terminal open inside one of the Docker containers, that can be faster as you can skip the "docker compose" step, but then not every Makefile target will be runnable inside a Docker container (like a target to rebuild the Docker image), so you have to awkwardly jump between terminals that are inside/outside the Docker container for different tasks? Any tricks here?