I feel like it's getting harder to tell what dagger is _actually_ for these days.
At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.
If I may offer a different perspective: Dagger has always been a general-purpose composition engine, built on container tech. Its most successful use case is CI - specifically taking complex build and test environments, and making them more portable and reproducible. But we never claimed to replace Jenkins or any other CI platform, and we've always been very open with our community about our desire to expand the use of Dagger beyond CI. We also never claimed to replace Docker, or to "be a shell" (note that the title of this HN page doesn't reflect the title of our post in that regard).
Every feature we ship is carefully designed for consistency with the overall design. For example, Dagger Shell is built on the same Dagger Engine that we've been steadily improving for years. It's just another client. Our goal is to build a platform that feels like Lego: each new piece makes all other pieces more useful, because they all can be composed together into a consistent system.
It's a fair point - My opinions and use case are my own, I didn't mean to imply or assume there were promises not kept. The dagger team has been nothing but supportive and I do think has built a great community.
That said, in the early days it was definitely pitched for CI/CD - and this how we've implemented it.
> What is it?
> Programmable: develop your CI/CD pipelines as code, in the same programming language as your application.
> Who is it for?
> A developer wishing your CI pipelines were code instead of YAML
Edit: This functionality/interaction with the dagger engine still exists today, and is what we rely on. The original comment is more of an observation on the new directions the project has taken since then.
I feel Nix is eating their lunch. Even though Nix as a language could be tough, it's a simpler solution than Dagger.
It provides a shell, almost complete reproducibility and isolation can be added through containers.
Working with dependencies to build source code using nix is usually straightforward but having the right binary version that doesn't have the same love as major languages (i.e. terraform, flux, etc) is one thing Nix really needs to implement. Marcelo's package version search[1] is a way to discover the particular hash but then you need to explicitly use that nixpkgs version for that particular binary and do a lot of mental gymnastics in your nix files. Nix could implement a syntax where you define the package versions as attribute set and then internally does a discovery of the nixpkgs hash for that versions and installs it. Flox and DevBox follow this pattern but I don't see why you need and external tool where this can be embedded in Nix (the cli).
We tried to fit dagger where we had jenkins - not just for binary builds, but for the other stuff. Mounting secrets for git clones / NPM installs, integration tests, terraform execution, build notifications and logging.
Caching is great, and dagger/nix both have interesting options here, but that's more of a bonus and not the core value prop of a build orchestrator.
I've found the Nix ecosystem to be lacking, missing packages, wrongly built packages (from the official upstream), and out of date versions. Homebrew still outclasses Nix in this regard (quality over quantity).
After taking Nix for a spin, I cannot be bothered to learn another custom tool with a bespoke language when I already have containers for doing the same things.
For Dagger, I can choose from a number of languages I already know and the Docker concepts map over nearly 1-1
Yup - Ever since the introduction of llm as a core API we're basically going to stop upgrading / using the project unless things start moving in a more sane direction.
This is very cool, I often find myself composing together a mixture of Dockerfiles and shell scripts for various images (build X Dockerfile for Y arch, save it as a base, build Z Dockerfile on top of the last thing, with W args). And then having to run it different ways with different mounts, based on if you're on a developer machine/robot/CI, running as a service/adhoc/attaching. This _looks_ like it could solve some of that messiness. I like that it can just reference the output of a build without having to muck around with tags (which are system global and can collide if you're not careful).
I have a particular hate on for the sandwich pattern where you have a Dockerfile to a "build" environment that then gets entered, and a bunch of stuff done in there, and then the results ADDed into a different Dockerfile for the publishable asset. I know multi-stage builds sort of address this need, but a lot of people aren't using them or have found cases where it's desirable to have the "running" context for the container, for example if a bunch of stuff is being volumed-in (which is now possible at build time but is janky).
Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers.
I don't like the sandwich either, it's too easy to accidentally miss a dependency. What I find myself doing more is building one Dockerfile as a base, and then doing `FROM $BASE` in the second to ease readability, caching, image pinning, etc.
> But then Packer has no concept of multi-stage or copying in assets from other containers.
I don't know about containers specifically, since I've never bothered to use packer for that process, but it does seem that packer supports multi-step artifact production and their example even uses docker to demonstrate it https://github.com/hashicorp/packer/blob/v1.9.5/website/cont...
OMG https://docs.dagger.io/features/debugging - Docker hasn't supported shelling into a broken build for a while now, this sounds so nice. I see there's some Dockerfile support, I wonder if I can just use this as a drop in replacement for `docker build` and get a nicer experience.
Dagger is built on the same underlying tech as docker build (buildkit). So the compatibility bridge is not a re-implementation of Dockerfile, it's literally the official upstream implementation.
Here's an example that 1) fetches a random git repo 2) builds from its dockerfile 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal:
git https://github.com/goreleaser/goreleaser |
head |
tree |
docker-build |
terminal |
publish ttl.sh/goreleaser-example-image
Being able to drop into failed container builds / runs has been game changing, saved me mucho time. If you are being programmatic, you can also inject the drop into interactive mode anywhere you want, without needing an error. You can use this to build up working environments into which you drop developers (kind of like dev containers)
The last time I tried something more than Buildx that Docker itself put out the experience was bad - something about not properly caching. I'll have to give this another shot sometime, though.
I had missed the fact that Dagger is apparently trying to... replace? Docker for container definitions. That's a pretty big vision.
I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling.
At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...
In theory you could replace much of Docker's low-level tooling with Dagger. In practice, that's not what we're trying to do. There's a lot of inertia to ripping out existing tools, and even if the replacement is better, it may still not be worth the effort.
What we're focusing on is green field application of container tech. Things that should be containerized, but aren't. For example:
- Cross-platform builds
- Complex integration testing environments
- Data processing pipelines
- AI agent workflows (where you give a LLM the ability to perform tasks in a controlled environment)
In those kinds of workflows, there is no dominant tool to replace - Docker or otherwise. Everyone builds their own unique snowflake of a monolith, by gluing together a dozen tools. Dagger aims to replace that glue with a modular, composable system.
The Bash compatible syntax is just one representation of an underlying cross-language API. You can also write code in your preferred language, provided there's an SDK for it. (Currently: Python, TypeScript, Go, PHP, Java, Elixir, Rust, .NET)
But if you're talking specifically about shell or a "native" REPL in a safer language - I also want to scratch that itch. Some day I'll dust off https://github.com/vito/dash :)
I feel this is much more powershell like than bash. The container thing looks much more than a typed object than a plain character device/pipe. Also what is the bourne again part?
Is it? Docker is quite long in the tooth and this point and is a long way from perfect. Docker's design is rife with poor choices and unaddressed flaws. One I've been tracking for years: there is no straightforward way to undo several things that get inherited by derived images, such as VOLUME and EXPOSE, and Docker Inc. doesn't care no matter how long the github threads get.
I see this more as a complement. I wouldn’t stop using Dockerfile for standalone definitions, but for shell+docker+shell sandwiches (the second shell being inside the container), this looks very handy.
i want to move away from GitHub Actions and I also want to move away from some bespoke monorepo tooling for something that's a better overall orchestrator of 'things'. Dagger scratches some of those itches, or seems to, so I've had my eye on it for a while. Knowing that they're trying to also help me define and use containerized environments - I sort of assumed I'd have to bring those myself.
I'm largely replacing Dockerfiles and Makefiles with a custom CLI that wraps Dagger. Being able to use a proper language, functions, packages, imports is so much better by comparison. (ymmv)
Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale.
I mean docker is really great but dagger in notebook formats just seems really cool ngl.
the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.
I would lump dagger in with JS (programming language, ecosystem, runtime) and MacOS (operating system) before comparing it to any single tool.
Answering "what does dagger do" can be tough, because there are very broad applications.
Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.
The article gives a few examples of concrete things you can do:
- Build, run, and publish containers
- Containerize language-specific builds
- Run integration tests in ephemeral environments, in a repeatable way
- Run your tools in controlled sandboxes with only the files, secrets and network access they need
It also mentions what you might replace with it:
> When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
I already build containers with Docker. I already have Makefiles that build containers and run commands in them. I already replace my complex shell scripts with Python scripts.
How does Dagger help? I just can't get anything concrete from their site.
Readit News
At first we'd hoped it could replace jenkins - it provided an alternative way to run and debug CI pipelines - right on your machine! You could write in golang and just import what you needed. The dev direction feels more scattered now, trying to replace docker, be a new shell(?), and weirdly trying to be some kind of langchain? Doing something different doesn't imply better. A new set of complicated CLI args is no better than what we started with (shell scripts, or jenkinsfiles to integrate docker builds). I'm a little bummed that the project has seemingly drifted (in my view) from the original mission.
Every feature we ship is carefully designed for consistency with the overall design. For example, Dagger Shell is built on the same Dagger Engine that we've been steadily improving for years. It's just another client. Our goal is to build a platform that feels like Lego: each new piece makes all other pieces more useful, because they all can be composed together into a consistent system.
That said, in the early days it was definitely pitched for CI/CD - and this how we've implemented it.
> What is it? > Programmable: develop your CI/CD pipelines as code, in the same programming language as your application.
> Who is it for? > A developer wishing your CI pipelines were code instead of YAML
https://github.com/dagger/dagger/blob/0620b658242fdf62c872c6...
Edit: This functionality/interaction with the dagger engine still exists today, and is what we rely on. The original comment is more of an observation on the new directions the project has taken since then.
[1] http://lazamar.github.io/download-specific-package-version-w...
We tried to fit dagger where we had jenkins - not just for binary builds, but for the other stuff. Mounting secrets for git clones / NPM installs, integration tests, terraform execution, build notifications and logging.
Caching is great, and dagger/nix both have interesting options here, but that's more of a bonus and not the core value prop of a build orchestrator.
After taking Nix for a spin, I cannot be bothered to learn another custom tool with a bespoke language when I already have containers for doing the same things.
For Dagger, I can choose from a number of languages I already know and the Docker concepts map over nearly 1-1
I think this approach of see what sticks and them trying out a lot of different things can be nice but not in its current form.
Although I have installed dagger to give it a try and I am just not sure how to make it work , the quickstart / hello world doesn't' work.
No I don't want to make an AI , I just want to see what you really are.
And so I do agree with your statement!
Or there's the copy my ansible playbooks into the container and run them there as localhost approach, which can be mitigated by what Packer does with at least running the playbook over SSH. But then Packer has no concept of multi-stage or copying in assets from other containers.
There's a lot of room to do better in this space.
I don't know about containers specifically, since I've never bothered to use packer for that process, but it does seem that packer supports multi-step artifact production and their example even uses docker to demonstrate it https://github.com/hashicorp/packer/blob/v1.9.5/website/cont...
Dagger is built on the same underlying tech as docker build (buildkit). So the compatibility bridge is not a re-implementation of Dockerfile, it's literally the official upstream implementation.
Here's an example that 1) fetches a random git repo 2) builds from its dockerfile 3) opens an interactive terminal to look inside 4) publish to a registry once you exit the terminal:
Dagger can read dockefiles as is, https://docs.dagger.io/cookbook#build-image-from-dockerfile
I really like the ambition here; it's very hard for me to translate this into belief that I could start using this now and actually replace some of my existing tooling.
At the same time, I kind of hate that they went for bash-compatible. I know everybody thinks bash is unbeatable, but at some point _surely_ we're going to move past its awful syntax and footguns (and SQL, and can I have a pony?)...
What we're focusing on is green field application of container tech. Things that should be containerized, but aren't. For example:
- Cross-platform builds
- Complex integration testing environments
- Data processing pipelines
- AI agent workflows (where you give a LLM the ability to perform tasks in a controlled environment)
In those kinds of workflows, there is no dominant tool to replace - Docker or otherwise. Everyone builds their own unique snowflake of a monolith, by gluing together a dozen tools. Dagger aims to replace that glue with a modular, composable system.
i think there's a decent chance we end up giving Dagger a spin this year.
But if you're talking specifically about shell or a "native" REPL in a safer language - I also want to scratch that itch. Some day I'll dust off https://github.com/vito/dash :)
Is it? Docker is quite long in the tooth and this point and is a long way from perfect. Docker's design is rife with poor choices and unaddressed flaws. One I've been tracking for years: there is no straightforward way to undo several things that get inherited by derived images, such as VOLUME and EXPOSE, and Docker Inc. doesn't care no matter how long the github threads get.
So I think there is ample room for alternatives.
Also what would you prefer to see instead of bash compatible?
https://github.com/nlewo/nix2container#nix2container
Dagger looks nifty, but it's too bad that it's still basically an imperative "start from thing, do stuff to it, publish" kind of model, rather than just expressing a desired end state and letting an underlying engine find its way there.
It's really neat, I recommend checking it out.
I am imagining this with a simple cloudflare tunnel and self hosting gitlab and I am really really seeing an open source way that developers can REALLY scale.
I mean docker is really great but dagger in notebook formats just seems really cool ngl.
the best part with the Dagger + Runme combo is that it runs entirely local. this isn't just a huge with for portability. it also cuts down development cycle times significantly.
> The cross-platform composition engine
> Build powerful software environments from modular components and simple functions. Perfect for complex builds and AI workflows.
This is generic to point of being useless.
Everything is a composition engine. Javascript is a composition engine. macOs is a composition engine.
Answering "what does dagger do" can be tough, because there are very broad applications.
Its always hard to describe general purpose platforms like these. Dagger is an open platform for building composable software. It has SDKs, sandboxed execution environment, observability, and now this shell interface.
Docker shell container - https://github.com/jrz/container-shell
"The devops operating system" does what?
- Build, run, and publish containers
- Containerize language-specific builds
- Run integration tests in ephemeral environments, in a repeatable way
- Run your tools in controlled sandboxes with only the files, secrets and network access they need
It also mentions what you might replace with it:
> When a workflow is too complex to fit in a regular shell, the next available option is often a brittle monolith: not as simple as a shell script, not as robust as full-blown software. The Dagger Shell aspires to help you replace that monolith with a collection of simple modules, composed with standard interfaces.
How does Dagger help? I just can't get anything concrete from their site.