I like to think of our solution as a modern version of a makefile. Many people are familiar with makefiles, and back before every CI server was configured using YAML and groovy, this was a common way to build things. We are trying to improve the simplicity of make by adding on the repeatability of docker containers. By mixing those two concepts together, a lot of cool properties fall out.
Because each target is containerized, there is no works-on-my-machine issues, or tests failing because something is not installed on one Jenkins runner or so on.
Second, that isolation means that dependencies between steps need to be explicitly declared, and with this knowledge, earthly can infer parallelism.
Third, containerization gives caching abilities. We can tell if nothing’s changed since a previous run, and earthly can cache steps across build runs or even across machines. You get to use a docker registry as a build cache.
We shared this on here about a year ago[1], and since that, lots has changed. First of all, a year ago it was just Vlad working on this, and now there is a number of us, including me. But for end-users, we have shipped many great features such as multi-arch builds, docker-compose support (which is handy for running integration tests ), inline and explicit caching, and an interactive debug mode. Interactive debug mode is neat but straightforward: a failing build step pops you into a shell at the point of failure.
I don't get it: why wouldn't I just use a Dockerfile? The Earthfiles look the same. Dockerfiles run the same way every time and Docker supports caching. I just see more complexity on top without value add.
If I want to move beyond local builds then I'd use something like Garden or Skaffold. Skaffold builds the same way every time and supports remote builders.
You can take a look at how the phoenix framework uses earthly to simplify their builds. They are testing across OTP versions and across different databases (mysql, SQL Server, postgres) all in an earthfile[1]. Ecto_sql does something similar[2].
I have a very hard time imagining "AWS Infinibuild(R)" that eats the "makefile + dockerfile that builds locally" use case, which is why I was curious to learn more
---
Separately, I would value GitHub's license detector gizmo using some of that sweet machine-learning sauce to identify the BSL, since it seems to be the path forward for these situations
Thank you for that; I would suggest putting that link in the PR description (or a comment) since not everyone who is similarly curious will be on a Show HN thread with the authoritative parties
I'm not trying to start trouble or get into a massive licensing thread on HN, but I want to point out that 2/5ths of those exemplar links of BSL success are in fact Apache 2.0 licensed, and not just source-available. I believe you that there are (evidently) parts of Kafka which one must pay money for, but my understanding is that the Apache Software Foundation would not tolerate hosting source-available projects on the apache.org domain nor in the apache GitHub organization.
Yesterday I gave up building something locally and just got CI to do it because to run locally you have to kind of make a poor mans .travis.yml file out of bash locally then play spot the difference (I never figured out what was different that was breaking it locally, resulting in a k8s deployment with a missing mount/volume).
This might sound weird but when you inherit some legacy code you’ll know what I mean.
Something where you can run the CI locally is such a boon.
Also CI providers do have downtime. If you can run locally, you can’t deploy your app!
Maybe this is a stupid question, but what is the "make" part? I see a lot of docker syntax and a few targets without dependencies. But make is all about dependencies. What kind of make dependencies do you support? What kind of variables, macros, etc?
We support dependencies in a number of ways, but are a bit different than make dependencies, because the file system of each step is separate unless you explicitly copy in or depend on other targets. This helps us do things in parallel because you are being explicit about the DAG of the build.
Copying or using file from a step causes your step to depend on it. Similarly for a `from +target`.
We also have variables, user defined commands and conditional logic. User defined commands are our way to composition.
I looked at their main page and got the impression this tool is not very / not at all useful - except perhaps for some niche I'm not in.
> Earthly is a syntax for defining your build.
I already have syntax for defining my build.
> It works with your existing build system.
If I have a build system, or a build system generator, why do I need to redefine my build another way?
> With Earthly, all builds are containerized
Huh? What? Why? ... That's already a deal breaker. I don't want to be limited to building within containers.
> With Earthly, all builds are ... repeatable
Isn't that true for any decent build system?
> With Earthly, all builds are ... language agnostic
An ambiguous statement. Especially since a build process does depend, to some extent, on the language of the source files.
> Get repeatable and understandable builds today.
(shrug) more marketing talk.
> It's like Makefile and Dockerfile had a baby.
I don't want Dockerfile in the family, and as for Makefile - it's too low-level. Makefiles get generated these days (and have been for over 20 years for most software packages, IIANM.)
> No more git commit -m "try again" .
This has never been a problem for me.
(Well, ok, if there's network trouble, then downloading things during a build will fail; but Earthly can't magically reconnect a broken network.)
> "Earthly is a pragmatic and incremental solution to a thorny problem that all organizations face"
Which thorny problem is that? The "non-repeatable builds"? I'm confused.
Earthly is essentially a make/dockerfile syntax for a BuildKit frontend. If you understand and agree that BuildKit has value, than that's the jist of it.
Agree with all your points for sure, but people need words on their landing page
I guess there's a niche for Mac/Windows developers to be able to run the build system on linux, because that's what their CI system runs (so it might fail for some issues that does not fail on their host machines), but most CI systems cannot run nested containers (because docker-in-docker requires some privileges), and as a result their CI system usually can not run this thing, which totally defeats the purpose.
> If I have a build system, or a build system generator, why do I need to redefine my build another way?
There are build systems such as SBT, Maven, Webpack etc, Other build systems such as Bazel completely replaces them, what earthly offers is integration with them.
The comparison to Bazel here is really not appropriate. The core competency of Bazel is to know exactly the build graph and be able to do the minimum amount of work possible. I don't see how this (basically syntactic sugar over multi-stage docker builds) is going to scale in a large monorepo with hundreds or thousands of developers and still maintain CI build times on the order of minutes.
If you're using, say, SBT and you like it, why do you need Earthly again? And if you don't like it, why don't you replace it if it's possible to replace?
I wonder what the scope of what would be a competitive service now that Earthly's license changed to BSL from MPL on 2021-02-01 [0]? Seems like Earthly Technologies Inc will launch a SaaS [1]. However, I'd assume it won't matter for most developers except for those selling a CI/CD SaaS. In any case, I'm looking forward to seeing what they end up offering!
Readit News
I like to think of our solution as a modern version of a makefile. Many people are familiar with makefiles, and back before every CI server was configured using YAML and groovy, this was a common way to build things. We are trying to improve the simplicity of make by adding on the repeatability of docker containers. By mixing those two concepts together, a lot of cool properties fall out.
Because each target is containerized, there is no works-on-my-machine issues, or tests failing because something is not installed on one Jenkins runner or so on.
Second, that isolation means that dependencies between steps need to be explicitly declared, and with this knowledge, earthly can infer parallelism.
Third, containerization gives caching abilities. We can tell if nothing’s changed since a previous run, and earthly can cache steps across build runs or even across machines. You get to use a docker registry as a build cache.
We shared this on here about a year ago[1], and since that, lots has changed. First of all, a year ago it was just Vlad working on this, and now there is a number of us, including me. But for end-users, we have shipped many great features such as multi-arch builds, docker-compose support (which is handy for running integration tests ), inline and explicit caching, and an interactive debug mode. Interactive debug mode is neat but straightforward: a failing build step pops you into a shell at the point of failure.
Let me know what you think of what we have built.
[1] https://news.ycombinator.com/item?id=22890612
If I want to move beyond local builds then I'd use something like Garden or Skaffold. Skaffold builds the same way every time and supports remote builders.
[1] https://phoenixframework.org/blog/improving-testing-and-cont...
[2] https://github.com/elixir-ecto/ecto_sql/blob/master/Earthfil...
There is also an easy to understand example with several targets with dependencies between them.
You can more or less achieve the same with multiple stages in a docker file, with named targets etc, but it get complicated fast.
Deleted Comment
the associated PR just says "doing it" without saying why https://github.com/earthly/earthly/pull/730
I have a very hard time imagining "AWS Infinibuild(R)" that eats the "makefile + dockerfile that builds locally" use case, which is why I was curious to learn more
---
Separately, I would value GitHub's license detector gizmo using some of that sweet machine-learning sauce to identify the BSL, since it seems to be the path forward for these situations
https://github.com/licensee/licensee/blob/master/docs/CONTRI...
I'm not trying to start trouble or get into a massive licensing thread on HN, but I want to point out that 2/5ths of those exemplar links of BSL success are in fact Apache 2.0 licensed, and not just source-available. I believe you that there are (evidently) parts of Kafka which one must pay money for, but my understanding is that the Apache Software Foundation would not tolerate hosting source-available projects on the apache.org domain nor in the apache GitHub organization.
I wish you all the best with your project
Deleted Comment
Rather, it looks like they’re calling Earthly “open core” or “source available” according to https://earthly.dev/blog/every-open-core-company-should-be-a...
> Free and open
Yesterday I gave up building something locally and just got CI to do it because to run locally you have to kind of make a poor mans .travis.yml file out of bash locally then play spot the difference (I never figured out what was different that was breaking it locally, resulting in a k8s deployment with a missing mount/volume).
This might sound weird but when you inherit some legacy code you’ll know what I mean.
Something where you can run the CI locally is such a boon.
Also CI providers do have downtime. If you can run locally, you can’t deploy your app!
Copying or using file from a step causes your step to depend on it. Similarly for a `from +target`.
We also have variables, user defined commands and conditional logic. User defined commands are our way to composition.
https://docs.earthly.dev/docs/guides/udchttps://docs.earthly.dev/docs/earthfile#copy
> Earthly is a syntax for defining your build.
I already have syntax for defining my build.
> It works with your existing build system.
If I have a build system, or a build system generator, why do I need to redefine my build another way?
> With Earthly, all builds are containerized
Huh? What? Why? ... That's already a deal breaker. I don't want to be limited to building within containers.
> With Earthly, all builds are ... repeatable
Isn't that true for any decent build system?
> With Earthly, all builds are ... language agnostic
An ambiguous statement. Especially since a build process does depend, to some extent, on the language of the source files.
> Get repeatable and understandable builds today.
(shrug) more marketing talk.
> It's like Makefile and Dockerfile had a baby.
I don't want Dockerfile in the family, and as for Makefile - it's too low-level. Makefiles get generated these days (and have been for over 20 years for most software packages, IIANM.)
> No more git commit -m "try again" .
This has never been a problem for me.
(Well, ok, if there's network trouble, then downloading things during a build will fail; but Earthly can't magically reconnect a broken network.)
> "Earthly is a pragmatic and incremental solution to a thorny problem that all organizations face"
Which thorny problem is that? The "non-repeatable builds"? I'm confused.
Agree with all your points for sure, but people need words on their landing page
I guess there's a niche for Mac/Windows developers to be able to run the build system on linux, because that's what their CI system runs (so it might fail for some issues that does not fail on their host machines), but most CI systems cannot run nested containers (because docker-in-docker requires some privileges), and as a result their CI system usually can not run this thing, which totally defeats the purpose.
> If I have a build system, or a build system generator, why do I need to redefine my build another way?
There are build systems such as SBT, Maven, Webpack etc, Other build systems such as Bazel completely replaces them, what earthly offers is integration with them.
If you're using, say, SBT and you like it, why do you need Earthly again? And if you don't like it, why don't you replace it if it's possible to replace?
[0] https://github.com/earthly/earthly/pull/730 [1] https://earthly.dev/tos