This computes a single hash based on the contents of all files. I suppose it's appropriate for languages/build systems that do not have file-level compilation. Otherwise, it would be more efficient to keep hashes per file and re-build only those that have changed.
A criticism on this implementation: The "state" directory is never cleaned; new states are always added. Therefore if you go back to a state (hash) that you had already built in the past, you will not be able to build it again.
And there is no need to create separate shell scripts, when you can have all the relevant code inside your Makefile. Presumably those are not going to be called independently anyway.
As I've written on previous discussions:
Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study...
Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
I wrote "the GNU Make Book has the code ready for you to study."
I'm currently traveling, without access to my books, but a quick online search at the contents of this book https://nostarch.com/download/GNU_Make_dTOC.pdf shows that p.82 has the code for "Rebuilding When a File's Checksum Changes".
The GNU Make Book by John Graham-Cumming, No Starch Press, April 2015, 256 pp., ISBN-13: 978-1-59327-649-2
Correct, the state directory isn't being cleared. As this is mostly aimed at making ephemeral build agents faster, this shouldn't matter much in practice.
Having said that, there is definitely some work to do on keeping the remote storage somewhat clean.
I paid full price for this book on No Starch, but I recently purchased it again as part of the "Linux by No Starch"[1] bundle that's going on now. I think that bundle would be appreciated by a lot of the crowd here. It's in the $10 and up tier.
Bravo! I haven't read the code carefully yet to fully understand the details of the implementation, but this is brilliant hackery.
As someone who has recently developed a focused interest on build systems, I have noticed the large vacuum in the space for a content-based Make-like that has a similarly low of a barrier of entry. HN user bobsomers said it well the other day:
"There is a much tighter, cleaner, simpler build system within Bazel struggling to get out. A judicious second take at it with a minimal focus, while taking some of the best ideas, could be wildly successful I think."
This is clever but truly content based change detection in make would seriously fix a bunch of issues. I'm sort of surprised it hasn't been done already.
I expect you could get rid of most of the non-POSIX features by using more complex recipes, probably exiling them into separate shell scripts. (The main limitation is that V7/POSIX/BSD-style suffix rules don’t let you specify a rule for producing ANYTHING.foo from ANYTHING, whereas V8/GNU-style pattern rules do.)
We use all kinds of tools that require a database that is effectively hidden from us. (e.g. git). I don't think this is a significant blocker or problem.
I might be misunderstanding but I think this fails when you revert to a previous revision of a file as the hashing will revert back to an older timestamp, i.e.
make
# compiled foo.ts
vim foo.ts
make
# compiled foo.ts
git stash
make
# nothing to do
Run all your scripts through https://www.shellcheck.net/ (you can install it locally too) and correct all errors it finds, click the explanation pages to understand why. In future, improve your style so you don't generate errors.
Readit News
A criticism on this implementation: The "state" directory is never cleaned; new states are always added. Therefore if you go back to a state (hash) that you had already built in the past, you will not be able to build it again.
And there is no need to create separate shell scripts, when you can have all the relevant code inside your Makefile. Presumably those are not going to be called independently anyway.
As I've written on previous discussions:
Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes. IIRC, the GNU Make Book has the code ready for you to study... Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
Can you point to some documentation for this? I haven't been able to find anything.
I'm currently traveling, without access to my books, but a quick online search at the contents of this book https://nostarch.com/download/GNU_Make_dTOC.pdf shows that p.82 has the code for "Rebuilding When a File's Checksum Changes".
The GNU Make Book by John Graham-Cumming, No Starch Press, April 2015, 256 pp., ISBN-13: 978-1-59327-649-2
Having said that, there is definitely some work to do on keeping the remote storage somewhat clean.
And (self promotion) it is in my book: https://nostarch.com/gnumake (pg 82)
[1]: https://www.humblebundle.com/books/linux-no-starch-press-boo...
As someone who has recently developed a focused interest on build systems, I have noticed the large vacuum in the space for a content-based Make-like that has a similarly low of a barrier of entry. HN user bobsomers said it well the other day:
"There is a much tighter, cleaner, simpler build system within Bazel struggling to get out. A judicious second take at it with a minimal focus, while taking some of the best ideas, could be wildly successful I think."
https://news.ycombinator.com/item?id=32831890
I think 'redo' is probably the closest contender these days.
apenwarr's redo implementation is probably the most popular, the most mature, and has been built with a lot of smart design choices:
https://redo.readthedocs.io/en/latest/
Pluggable caching, which AFAIK apenwarr/redo does not yet support, would be a great addition, to support remote, shared caching.
[1]: https://gittup.org/tup/
1. I find the syntax pretty offputting. 2. Make is found everywhere so a drop in replacement is a useful feature.
It could use xattr to store its content tags.
The main use case for this is ephemeral build hosts to share the cache, but I want to work all these issues out too, so thank you for the feedback
- https://www.cmcrossroads.com/article/rebuilding-when-files-c... - http://olipratt.co.uk/rebuilding-makefile-targets-only-when-...
Wait what
Oh, just a typo.Run all your scripts through https://www.shellcheck.net/ (you can install it locally too) and correct all errors it finds, click the explanation pages to understand why. In future, improve your style so you don't generate errors.
Here's some more I've found useful: - https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html - https://tldp.org/LDP/abs/html/index.html