Dagger was something I looked into two or so years ago before they got consumed by the LLM and AI agent hype, and while the promise of being able to run the exact CI workflows locally seemed excellent, it seemed that there's basically no way be a Dagger user without buying into their Dagger Cloud product.
I ended up opting for CUE and GitHub Actions, and I'm glad I did as it made everything much, much simpler.
Can you explain/link to why you can't really use this without their cloud product? I'm not seeing anything at a glance, and this looks useful for a project of mine, but I don't want to be trapped by limitations that I only find out about after putting in weeks of work
Overall I like Dagger conceptually, but I wish they'd start focusing more on API stability and documentation (tbf it's not v1.0). v0.19 broke our Dockerfile builds and I don't feel like figuring out the new syntax atm. Having to commit dev time to the upgrade treadmill to keep CI/CD working was not the dream.
re: the cloud specifically see these GitHub issues:
Basically if you want consistently fast cached builds it's a PITA and/or not possible without the cloud product, depending on how you set things up. We do run it self-hosted though, YMMV.
Hi, I'm the founder of Dagger. It's not true that you can't use Dagger without our cloud offering. At the moment our only commercial product is observability for your Dagger pipelines. It's based on standard otel telemetry emitted by our open source engine. It's completely optional.
If you have questions about Dagger, I encourage you to join our Discord server, we will be happy to answer them!
> If you’ve been active in the Dagger community, this news will come as no surprise. Since we released multi-language support, we have seen a steep decline in usage of our original CUE configuration syntax, and have made it clear that feature parity with newer SDKs would not be a priority.
That is, of course, a self-fulfilling prophecy (or, perhaps, a self-inflicted wound). As soon as Dagger's "multi-language support" came out (actually a bit before), the CUE SDK was rendered abandonware. Development only happened on the new backend, and CUE support was never ported over to the new one.
What I don't get is why would someone code in the terrible GitHub actions dsl which only runs on GitHub actions and nowhere else when there are so many other options that run perfectly fine if you just run it from GitHub actions.
When I got started it was much more difficult as you had to do a lot of manual work to get things started, and you really had to believe the promises that CUE offered (which I did...), but nowadays they've made so many steps in the right direction that getting something going is far quicker!
As someone that has used Dagger a lot (a previous daggernaut / ambassador dropped off after LLMs was announced, and was changing jobs at the time. implemented it at a previous company across 95% of services, built the rust sdk) the approach was and is amazing for building complex build chains.
It serves a place where a dockerfile is not enough, and CI workflows are too difficult to debug or reason about.
I do have some current problems with it though:
1. I don't care at all about the LLM agent workflows, I get that it is possible, but the same people that chose dagger for what it was, is not the same audience that runs agents like that. I can't choose dagger currently, because I don't know if they align with my interests as an engineer solving a specific problems for where I work (delivering software, not running agents).
2. I advocated for modules before it was a thing, but I never implemented it. It is too much magic, I want to write code, not a DSL that looks like code, dagger is already special in that regard, to modules takes it a step too far. You can't find the code in their docs anymore, but dagger can be written with just a .go, .py or .rs file. Simply take in dagger as a dependency and build your workflow.
3. Too complex to operate, dagger doesn't have runners currently, and it is difficult to run a setup in production for CI yourself, without running it in the actions themselves, which can be disastrous for build times, as dagger often leads you into using quite a few images, so having a cache is a must.
Dagger needs to choose and execute; not having runners, even when we we're willing to throw money at them was a mistake IMO. Love the tool, the team, the vision but it is too distracted, magical and impatient to pick up at the moment.
Hi Kasper, good to see you here! Thank you for the detailed feedback.
1. Yes we got over-excited with the agent runtime use case. We stand by the LLM implementation because we never compromised on the integrity of Dagger's modular design. But our marketing and product priorities were all over the place. We're going to refocus on the original use case: helping you ship software, and more particularly building & testing it.
2. Modules have warts but they are essential. We will continue to improve them, and remain committed to them. Without this feature, you have to write a complete standalone program every time you want to build or test your software. It's too much overhead.
3. Yes you are right. We really thought we could co-exist with CI runners, and get good performance without reinventing the wheel. But for various reasons that turned out to not be the case. So we're going to ship vertically integrated runners, with excellent scalability and performance. DM me if you want early access :)
TLDR: yes we needed to choose and execute. We have, and we are.
I thought Dagger had/has a lot of potential to be "AWS-CDK for CI pipelines".
I.e. declaratively setup a web of CI / deployment tasks, based on docker, with a code-first DSL, instead of the morass of copy-pasted (and yes orbs) CircleCI yaml files we have strewn about our internals repos.
But their DSL for defining your pipelines is ... golang? Like who would pick golang as "a friendly language for setting up configs".
The underlying tech is technically language-agnostic, just as aws-cdk's is (you can share cdk constructs across TypeScript/Python), but it's rooted in golang as the originating/first-class language, so imo will never hit aws-cdk levels of ergonomics.
That technical nit aside, I love the idea; ran a few examples of it a year or so ago and was really impressed with the speed; just couldn't wrap my around "how can I make this look like cdk".
Still looks like "a circa-2000s Java builder API" and doesn't look like pleasant / declarative / idiomatic TypeScript, which is what aws-cdk pulled off.
Genuinely impressively (imo), aws-cdk intermixes "it's declarative" (you're setting up your desired state) but also "it's code" (you can use all the usual abstractions) in a way that is pretty great & unique.
I was interested in the beginning for CI/CD, but then they tried to take a kind of "AI-oriented" view in order to ride the AI wave, and the value prop of their tool was completely muddied up...
Hi, I'm the founder of Dagger. I can't speak to the negativity, but if you're looking for a way to make your CI more portable, I recommend joining our Discord and asking our community directly about the pros and cons of using Dagger. Even if you don't end up using it, there are a lot of people there who are passionate about CI and can recommend other alternatives, in a more constructive and pragmatic way than you are getting here.
I used the old CUE-based version when it came out, and was really excited about it. I liked it, and enjoyed working with CUE, but the API was clunky and incomplete.
Then they completely abandoned not just the CUE frontend, but CUE altogether (while strenuously denying that they were doing so) for a GraphQL-based rewrite that focused on letting people use popular general-purpose languages to construct their workflows. The initial rollout of this was not feature complete and only supported imperative languages (Python and TypeScript, IIRC), which I didn't like.
Instead of porting everything over to all their new interfaces, I hopped off the train and rewrote all of our portable pipeline scripts in Nix, via Devenv. At the time, I'd never used Devenv before, but getting the work done that time still took maybe a tenth of the time or less. More than anything else, this was due to not having to fuck around with the additional overhead Docker entails (fussing with mount points, passing files from one stage to another, rebuilding images, setting up VMs... all of it). I got the reproducibility without the extra bullshit, and got to work with interfaces that have proven much more stable.
I still think there's a place for something like Dagger, focused just on CI, perhaps even still using Docker as a deployment/distribution strategy. But I no longer trust Dagger to execute on that. I think a proper external DSL (probably special-purposw but still Turing-complete, e.g., Nickel) is the right fit for this domain, and that it should support multiple means of achieving repeatability rather than just Docker (e.g., Nix on bare metal and Podman, to start). An option to work on bare metal via reproducible environment management tools like Nix, Guix, or Spack is a valuable alternative to burdensome approaches like containers.
I haven't looked at Dagger in several months, but the other big piece that is missing for portable CI workflows is a library that abstracts over popular CI platforms so you can easily configure pull/merge request pipelines without worrying about the implementation details like what environment variables each platform exposes to indicate source and target branch.
Idk anything about all the AI horseshit; I was off the Dagger bandwagon before they took that turn. I don't know if it's serious or a nominal play to court investors. But that kind of pivot is another reason not to build core infra on top of the work of startups imo. If the product is 70% of what you want, you have no way of knowing whether filling that 30% gap is something the maintainers will suddenly pivot away from, even if their current direction looks aligned with yours.
I'd recommend considering tools in this space only if (a) they're already close to 100% of what you need and (b) they're open-source. Maybe you can relax (a) if it's really easy to extend the codebase (I find this to be true for Devenv's Nix modules, for example.)
Do you have any examples of your Devenv workflow you can share? I took a look at Dagger and really like the concept, but I'm trying to figure out the limitations/why there's so much negativity in this thread.
I currently manage my development environments via NixOS and Devenv, so if I could just keep that and achieve the same functionality, that sounds good to me.
Readit News
I ended up opting for CUE and GitHub Actions, and I'm glad I did as it made everything much, much simpler.
re: the cloud specifically see these GitHub issues:
https://github.com/dagger/dagger/issues/6486
https://github.com/dagger/dagger/issues/8004
Basically if you want consistently fast cached builds it's a PITA and/or not possible without the cloud product, depending on how you set things up. We do run it self-hosted though, YMMV.
If you have questions about Dagger, I encourage you to join our Discord server, we will be happy to answer them!
That is, of course, a self-fulfilling prophecy (or, perhaps, a self-inflicted wound). As soon as Dagger's "multi-language support" came out (actually a bit before), the CUE SDK was rendered abandonware. Development only happened on the new backend, and CUE support was never ported over to the new one.
https://news.ycombinator.com/item?id=46262846
Here are a few links to whet your appetite:
- https://cue.dev/docs/getting-started-with-github-actions-cue...
- https://cue.dev/docs/drying-up-github-actions-workflows/
- https://cue.dev/docs/spotting-errors-earlier-github-actions-...
Definitely read through the CUE documentation (https://cuelang.org/docs/), watch their YouTube videos (https://www.youtube.com/@cuelang/videos), and join the community Slack channel (https://cuelang.org/community/). I've gotten a lot of help in the Slack from both enthusiastic community members and from the developers themselves whenever I've gotten stuck.
It serves a place where a dockerfile is not enough, and CI workflows are too difficult to debug or reason about.
I do have some current problems with it though:
1. I don't care at all about the LLM agent workflows, I get that it is possible, but the same people that chose dagger for what it was, is not the same audience that runs agents like that. I can't choose dagger currently, because I don't know if they align with my interests as an engineer solving a specific problems for where I work (delivering software, not running agents).
2. I advocated for modules before it was a thing, but I never implemented it. It is too much magic, I want to write code, not a DSL that looks like code, dagger is already special in that regard, to modules takes it a step too far. You can't find the code in their docs anymore, but dagger can be written with just a .go, .py or .rs file. Simply take in dagger as a dependency and build your workflow.
3. Too complex to operate, dagger doesn't have runners currently, and it is difficult to run a setup in production for CI yourself, without running it in the actions themselves, which can be disastrous for build times, as dagger often leads you into using quite a few images, so having a cache is a must.
Dagger needs to choose and execute; not having runners, even when we we're willing to throw money at them was a mistake IMO. Love the tool, the team, the vision but it is too distracted, magical and impatient to pick up at the moment.
1. Yes we got over-excited with the agent runtime use case. We stand by the LLM implementation because we never compromised on the integrity of Dagger's modular design. But our marketing and product priorities were all over the place. We're going to refocus on the original use case: helping you ship software, and more particularly building & testing it.
2. Modules have warts but they are essential. We will continue to improve them, and remain committed to them. Without this feature, you have to write a complete standalone program every time you want to build or test your software. It's too much overhead.
3. Yes you are right. We really thought we could co-exist with CI runners, and get good performance without reinventing the wheel. But for various reasons that turned out to not be the case. So we're going to ship vertically integrated runners, with excellent scalability and performance. DM me if you want early access :)
TLDR: yes we needed to choose and execute. We have, and we are.
Thank you again for the feedback.
Best of luck and thx for taking my harsh feedback in strides!
I.e. declaratively setup a web of CI / deployment tasks, based on docker, with a code-first DSL, instead of the morass of copy-pasted (and yes orbs) CircleCI yaml files we have strewn about our internals repos.
But their DSL for defining your pipelines is ... golang? Like who would pick golang as "a friendly language for setting up configs".
The underlying tech is technically language-agnostic, just as aws-cdk's is (you can share cdk constructs across TypeScript/Python), but it's rooted in golang as the originating/first-class language, so imo will never hit aws-cdk levels of ergonomics.
That technical nit aside, I love the idea; ran a few examples of it a year or so ago and was really impressed with the speed; just couldn't wrap my around "how can I make this look like cdk".
https://docs.dagger.io/cookbook/services?sdk=typescript
Still looks like "a circa-2000s Java builder API" and doesn't look like pleasant / declarative / idiomatic TypeScript, which is what aws-cdk pulled off.
Genuinely impressively (imo), aws-cdk intermixes "it's declarative" (you're setting up your desired state) but also "it's code" (you can use all the usual abstractions) in a way that is pretty great & unique.
What else could be used to abstract away your CICD from the launcher (Jenkins, Argo Workflows, GitHub Actions, etc.)?
Then they completely abandoned not just the CUE frontend, but CUE altogether (while strenuously denying that they were doing so) for a GraphQL-based rewrite that focused on letting people use popular general-purpose languages to construct their workflows. The initial rollout of this was not feature complete and only supported imperative languages (Python and TypeScript, IIRC), which I didn't like.
Instead of porting everything over to all their new interfaces, I hopped off the train and rewrote all of our portable pipeline scripts in Nix, via Devenv. At the time, I'd never used Devenv before, but getting the work done that time still took maybe a tenth of the time or less. More than anything else, this was due to not having to fuck around with the additional overhead Docker entails (fussing with mount points, passing files from one stage to another, rebuilding images, setting up VMs... all of it). I got the reproducibility without the extra bullshit, and got to work with interfaces that have proven much more stable.
I still think there's a place for something like Dagger, focused just on CI, perhaps even still using Docker as a deployment/distribution strategy. But I no longer trust Dagger to execute on that. I think a proper external DSL (probably special-purposw but still Turing-complete, e.g., Nickel) is the right fit for this domain, and that it should support multiple means of achieving repeatability rather than just Docker (e.g., Nix on bare metal and Podman, to start). An option to work on bare metal via reproducible environment management tools like Nix, Guix, or Spack is a valuable alternative to burdensome approaches like containers.
I haven't looked at Dagger in several months, but the other big piece that is missing for portable CI workflows is a library that abstracts over popular CI platforms so you can easily configure pull/merge request pipelines without worrying about the implementation details like what environment variables each platform exposes to indicate source and target branch.
Idk anything about all the AI horseshit; I was off the Dagger bandwagon before they took that turn. I don't know if it's serious or a nominal play to court investors. But that kind of pivot is another reason not to build core infra on top of the work of startups imo. If the product is 70% of what you want, you have no way of knowing whether filling that 30% gap is something the maintainers will suddenly pivot away from, even if their current direction looks aligned with yours.
I'd recommend considering tools in this space only if (a) they're already close to 100% of what you need and (b) they're open-source. Maybe you can relax (a) if it's really easy to extend the codebase (I find this to be true for Devenv's Nix modules, for example.)
I currently manage my development environments via NixOS and Devenv, so if I could just keep that and achieve the same functionality, that sounds good to me.
Then... it wasn't. The more I read the less I ever want to see this again. The LLM train has got to end at some point.