Hi, I work on Dart and was one of the people working on this feature. Reposting my Reddit comment to provide a little context:
I'm bummed that it's canceled because of the lost time, but also relieved that we decided to cancel it. I feel it was the right decision.
We knew the macros feature was a big risky gamble when we took a shot at it. But looking at other languages, I saw that most started out with some simple metaprogramming feature (preprocessor macros in C/C++, declarative macros in Rust, etc.) and then later outgrew them and added more complex features (C++ template metaprogramming, procedural macros in Rust). I was hoping we could leapfrog that whole process and get to One Metaprogramming Feature to Rule Them All.
Alas, it is really hard to be able to introspect on the semantics of a program while it is still being modified in a coherent way without also seriously regressing compiler performance. It's probably not impossible, but it increasingly felt like the amount of work to get there was unbounded.
I'm sad we weren't able to pull it off but I'm glad that we gave it a shot. We learned a lot about the problem space and some of the hidden sharp edges.
I'm looking forward to working on a few smaller more targeted features to deal with the pain points we hoped to address with macros (data classes, serialization, stateful widget class verbosity, code generation UX, etc.).
Sounds like performance is the biggest issue. I'm guessing the macros need to run after every keystroke. That creates big time constraint that while nice it's not really needed.
Due to AOT compilation, some form of (pre)compile time code generation is needed, but it doesn't need to be macros. It doesn't need to be instantaneous, but it also shouldn't take minutes.
Adding features directly into the language removes the need for some code generation.
Augmentations will already make code generation much nicer to use.
build_runner needs to become more integrated so that IDEs would read build_runner's config and run it automatically.
IMO it would make the most sense if code generation was integral part of the dart build system. Few years ago I described shortcomings of build_runner on github https://github.com/flutter/flutter/issues/63323, not much changed afaik.
Would you happen to have any feel for / comparative docs around how Java does all this? Java has had compile-time metaprogramming for a very long time, integrated with IDEs (well. eclipse), the type system, hot reloading, AOT, etc. It seems like one worth looking closely at, since most are rather simple in comparison (e.g. a huge amount are syntax-based and don't have in-progress type information).
Though it would not surprise me at all if it was considered too slow to adopt. I don't remember compile-time performance issues with it when I did Java work, but I was definitely not paying close attention, and the ecosystem strongly prefers runtime shenanigans in general.
I always feel better about the stewardship of a project when you see a thoughtfully written reason for saying no to a feature, especially when there’s already sunk cost. Props to the team.
This is good news. The dart language has been getting more complicated without corresponding quality of life improvements. A first class record object without messing around with macros would be a great start.
Because implementing them is tedious, and you can always simulate them with simpler aggregation methods, or possibly lexical closures.
When the language implementors start making larger programs, it will soon become apparent how the program organization is hampered without named, defined data structures.
I didn't add structs to TXR Lisp until August 2015, a full six years from the start of the project. I don't remember it being all that much fun, except when I changed my mind about single inheritance and went multiple. The first commit for that was in December 2019.
Another fun thing was inventing a macro system for defstruct, allowing new kinds of clauses to be written that can be used inside defstruct. Then using them to write a delegation mechanism in the form of :delegate and :mass-delegate clauses, whereby you can declare individual methods, or a swath of them, to delegate through another object.
Because it's arguably syntactic sugar and, IMHO, it's worked out better for developers for Dart to model it as a 3rd party library problem. i.e. have a JSONSerializable protocol, and enable easy code generation by libraries.
i.e. I annotate my models with @freezed, which also affords you config of the N different things people differ on with json (are fields snake case? camel case? pascal case?) and if a new N+1 became critical, I could hack it in myself in 2-4 hours.
I'm interested to see how this'd integrate with the language while affording the same functionality. Or maybe that's the point: it won't, but you can always continue using the 3rd party libraries. But now it's built into the language, so it is easier to get from 0 to 1.
I think after reading through the blog post the reasons they have made a whole lot of sense and sounded like that of a mature engineering team to me.
There are a bunch of other interesting approaches here they can look at. Improving the code generation story more generally, shopping the augmentations feature (basically C#’s partial classes) and getting more serious about serialization all feel like sensible directions from here.
There is a really interesting community proposal at the moment on the serialization front that I think would solve a lot of the issues that got people so excited about macros in the first place here: https://github.com/schultek/codable/blob/main/docs/rfc.md
This sounds like they were going for a Roslyn analogue (using Dart to generate Dart the same way Roslyn uses C# to generate C#). Definitely a big time investment.
It's a big bite to chew, but I think Roslyn has paid big dividends.
Macros give their own kind of power, and it's a tough call to give that up for runtime hot-reloading. Languages like Haxe have macros, but also have hot reloading capabilities that typically are supported in certain game frameworks. You probably don't want to mix them together, but it's also a good development process to have simpler compilation targets that enable more rapid R&D, and then save macros for larger/more comprehensive builds.
> Runtime introspection (e.g., reflection) makes it difficult to perform the tree-shaking optimizations that allow us to generate smaller binaries.
Does anyone have any more information on How Dart actually does Tree Shaking? And what is "Tree Shakeable"? This issue is still open on Github https://github.com/Dart-lang/sdk/issues/33920.
I think this quote accurately sums things up
> In fact the only references I can find anywhere to this feature is on the Dart2JS page:
> Don’t worry about the size of your app’s included libraries. The dart2js tool performs tree shaking to omit unused classes, functions, methods, and so on. Just import the libraries you need, and let dart2js get rid of what you don’t need.
> This has led customers to wild assumptions around what is and what is not tree-shakeable, and without any clear guidance to patterns that allow or disallow tree-shaking. For example internally, many large applications chose to store configurable metadata in a hash-map:
I don't have a full answer for you, but I know a little. I've hacked on the Dart compiler some, but my relationship with Dart has mostly been as a creator of Flutter and briefly Eng Dir for the Dart project.
Dart has multiple layers where it does tree shaking.
The first one is when building the "dill" (dart intermediate language) file, which is essentially the "front-end" processing step of the compiler which takes .dart files and does amount of processing. At that step things like entire unused libraries and classes are removed I believe.
When compiling to an ahead of time compiled binary (e.g. for releasing to iOS or Android) Dart does additional steps where it collects a set of roots and walks from those roots to related objects in the graph and discards all the rest. Not unlike a garbage collection. There are several passes of this for different parts of the compile, including as Dart is even writing the binary it will drop things like class names for unused classes (but keep their id in the snapshot so as not to re-number all the other classes).
It is incredibly slow though. I have a project with 40k lines of code which takes a minute to generate on an m1. It's a far cry from incremental compilation. It's enough that I generally avoid adding anything new that would require generation.
I agree, Dart's public-facing codegen system (build_runner) leaves a lot to be desired. (In part the problem is that Dart uses a separate system inside Google.)
However, this is a topic of active work for the Dart team: https://github.com/dart-lang/build/issues/3800. I'm sure they would welcome your feedback, particularly if you have examples you can share.
You're also always welcome to reach out to me if you have Flutter/Dart concerns. I founded the Flutter project (and briefly led the Dart team) and care a great deal about customer success with both. eric@shorebird.dev reaches me.
You should be able to leverage generate_for in your build.yaml with include/exclude to reduce those build times significantly. You should be able to get it back down to a few seconds including building the graph and then you should be able to just run watch instead of build.
It may be worth mentioning that build_runner's graph contains every single asset that might be generated. So when selecting what's included and excluded you can reduce the graph size dramatically.
It takes a minute to build from scratch or to update when running "build_runner watch"? My app is over 40k lines and watch updates almost instantaneously.
Thanks, good take. Especially if some of the pieces are still being added.
In my humble opinion, you can handle many cases like serialization better with 'compileTime' or comptime features though I'm partial to macros. Especially with core compile time constructs like 'fields' [1, 2]. Though those require some abilities dart's compiler may not have or be able to do efficiently. That'd be a bummer, as even C++ is finally getting compile time reflection.
Readit News
I'm bummed that it's canceled because of the lost time, but also relieved that we decided to cancel it. I feel it was the right decision.
We knew the macros feature was a big risky gamble when we took a shot at it. But looking at other languages, I saw that most started out with some simple metaprogramming feature (preprocessor macros in C/C++, declarative macros in Rust, etc.) and then later outgrew them and added more complex features (C++ template metaprogramming, procedural macros in Rust). I was hoping we could leapfrog that whole process and get to One Metaprogramming Feature to Rule Them All.
Alas, it is really hard to be able to introspect on the semantics of a program while it is still being modified in a coherent way without also seriously regressing compiler performance. It's probably not impossible, but it increasingly felt like the amount of work to get there was unbounded.
I'm sad we weren't able to pull it off but I'm glad that we gave it a shot. We learned a lot about the problem space and some of the hidden sharp edges.
I'm looking forward to working on a few smaller more targeted features to deal with the pain points we hoped to address with macros (data classes, serialization, stateful widget class verbosity, code generation UX, etc.).
Due to AOT compilation, some form of (pre)compile time code generation is needed, but it doesn't need to be macros. It doesn't need to be instantaneous, but it also shouldn't take minutes.
Adding features directly into the language removes the need for some code generation.
Augmentations will already make code generation much nicer to use.
build_runner needs to become more integrated so that IDEs would read build_runner's config and run it automatically.
Though it would not surprise me at all if it was considered too slow to adopt. I don't remember compile-time performance issues with it when I did Java work, but I was definitely not paying close attention, and the ecosystem strongly prefers runtime shenanigans in general.
In practice in a Dart app you usually use freezed or something similar: https://pub.dev/packages/freezed
When the language implementors start making larger programs, it will soon become apparent how the program organization is hampered without named, defined data structures.
I didn't add structs to TXR Lisp until August 2015, a full six years from the start of the project. I don't remember it being all that much fun, except when I changed my mind about single inheritance and went multiple. The first commit for that was in December 2019.
Another fun thing was inventing a macro system for defstruct, allowing new kinds of clauses to be written that can be used inside defstruct. Then using them to write a delegation mechanism in the form of :delegate and :mass-delegate clauses, whereby you can declare individual methods, or a swath of them, to delegate through another object.
i.e. I annotate my models with @freezed, which also affords you config of the N different things people differ on with json (are fields snake case? camel case? pascal case?) and if a new N+1 became critical, I could hack it in myself in 2-4 hours.
I'm interested to see how this'd integrate with the language while affording the same functionality. Or maybe that's the point: it won't, but you can always continue using the 3rd party libraries. But now it's built into the language, so it is easier to get from 0 to 1.
There are a bunch of other interesting approaches here they can look at. Improving the code generation story more generally, shopping the augmentations feature (basically C#’s partial classes) and getting more serious about serialization all feel like sensible directions from here.
There is a really interesting community proposal at the moment on the serialization front that I think would solve a lot of the issues that got people so excited about macros in the first place here: https://github.com/schultek/codable/blob/main/docs/rfc.md
It's a big bite to chew, but I think Roslyn has paid big dividends.
https://haxe.org/manual/macro.html
https://github.com/RblSb/KhaHotReload
Does anyone have any more information on How Dart actually does Tree Shaking? And what is "Tree Shakeable"? This issue is still open on Github https://github.com/Dart-lang/sdk/issues/33920.
I think this quote accurately sums things up
> In fact the only references I can find anywhere to this feature is on the Dart2JS page:
> Don’t worry about the size of your app’s included libraries. The dart2js tool performs tree shaking to omit unused classes, functions, methods, and so on. Just import the libraries you need, and let dart2js get rid of what you don’t need.
> This has led customers to wild assumptions around what is and what is not tree-shakeable, and without any clear guidance to patterns that allow or disallow tree-shaking. For example internally, many large applications chose to store configurable metadata in a hash-map:
Dart has multiple layers where it does tree shaking.
The first one is when building the "dill" (dart intermediate language) file, which is essentially the "front-end" processing step of the compiler which takes .dart files and does amount of processing. At that step things like entire unused libraries and classes are removed I believe.
When compiling to an ahead of time compiled binary (e.g. for releasing to iOS or Android) Dart does additional steps where it collects a set of roots and walks from those roots to related objects in the graph and discards all the rest. Not unlike a garbage collection. There are several passes of this for different parts of the compile, including as Dart is even writing the binary it will drop things like class names for unused classes (but keep their id in the snapshot so as not to re-number all the other classes).
I have no experience with tree shaking in the dart2js compiler, but there are experts on Discord who might be able to answer: https://github.com/flutter/flutter/blob/master/docs/contribu...
What exactly all this means as a dev using Dart, I don't know. In general I just assume the tree shaking works and ignore it. :)
The Dart tech lead has done some writings, but none seem to cover the exact details of treeshaking: https://mrale.ph/dartvm/https://github.com/dart-lang/sdk/blob/main/runtime/docs/READ...
However, this is a topic of active work for the Dart team: https://github.com/dart-lang/build/issues/3800. I'm sure they would welcome your feedback, particularly if you have examples you can share.
You're also always welcome to reach out to me if you have Flutter/Dart concerns. I founded the Flutter project (and briefly led the Dart team) and care a great deal about customer success with both. eric@shorebird.dev reaches me.
It may be worth mentioning that build_runner's graph contains every single asset that might be generated. So when selecting what's included and excluded you can reduce the graph size dramatically.
I'd like to believe this is a good thing for the Dart project, but only time will tell. My hot take here: https://shorebird.dev/blog/dart-macros/
In my humble opinion, you can handle many cases like serialization better with 'compileTime' or comptime features though I'm partial to macros. Especially with core compile time constructs like 'fields' [1, 2]. Though those require some abilities dart's compiler may not have or be able to do efficiently. That'd be a bummer, as even C++ is finally getting compile time reflection.
1: https://nim-lang.org/docs/iterators.html#fieldPairs.i%2CT 2: https://www.openmymind.net/Basic-MetaProgramming-in-Zig/