Starlark is definitely a mixed experience IMO, from my 7 years working with it in Bazel
On one hand, having a "hermetic" subset of Python is nice. You can be sure your Bazel starlark codebase isn't going to be making network calls or reading files or shelling out to subprocesses and all that. The fact that it is hermetic does help make things reproducible and deterministic, and enables paralleization and caching and other things. Everyone already knows Python syntax, and its certainly nicer than the templated-bash-in-templated-yaml files common elsewhere in the build tooling/infra space
On the other hand, a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti. Starlark goes the extra mile to be non-turing-complete, but that doesn't mean it's performant or easy to understand. And starlark's minimalism is also a curse as it lacks many features that help you manage large Python codebases such as PEP484 type annotations, which also means IDEs also cannot provide much help since they rely on types to understand the code
For https://mill-build.org we went the opposite route: not enforcing purity, but using a language with strong types and a strong functional bent to it. So far it's been working out OK, but it remains to be seen how well it scales to ever larger and more complex build setups
I have been building an internal tools development and deployment platform [1]. It is built in Go, using Starlark for configuration and API business logic.
Starlark has been great to build with. You get the readability of having a simple subset of python, without python's dependency management challenges. It is easily extensible with plugin APIs. Concurrent API performance is great without the python async challenges.
One challenge wrt using Starlark as an general purpose embedded scripting language is that it does not support usual error handling features. There are no exceptions and no multi value return for error values, all errors result in an abort. This works for a config language, where a fail-fast behavior is good. But for a general purpose script, you need more fine grained error handling. Since I am using Starlark for API logic only, I came up with a user definable error handling behavior. This uses thread locals to keep track of error state [2], which might not work for more general purpose scripting use cases.
A lot of the complaints about Starlark as a programming language, and the proposed alternatives, seem to me to miss out on the UX advantages of having pythonic scripting (which so many folks who have taken a random "coding" class understand intuitively) whereas, e.g., using a lisp or lua would not. Further, having a language and runtime designed for safe use is absolutely critical, and trying to embed another runtime (js/wasm) and manage to lock it down successfully, is a much larger undertaking than I think folks realize.
This is also one of my major complaints at this point. I'm building a developer tool with starklark resting at its core and I had already came across your work.
I wish the starklark team had addressed it at this point.
> a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti
This brings it to the point. I'm still wondering why the achievements of software engineering of the past fifty years, like modularization and static type checking had apparently so little influence on build systems. I implemented https://github.com/rochus-keller/BUSY for this reason, but it seems to require a greater cultural shift than most developers are already willing to make.
It did have influence. Take a look at Gradle, which is widely used in the JVM space. It uses a general, strongly typed language (Kotlin) to configure it and it has a very sophisticated plugin and modules system for the build system itself, not just for the apps it's building.
Gradle has its problems, and I often curse it for various reasons, but I'm pretty glad it uses regular languages that I can reuse in non-build system contexts. And the fact that it just bites the bullet and treats build systems as special programs with all the same support that it gives to the programs it's building does have its advantages, even if the results can get quite complex.
Build systems are sort of like type expressions, templates, or constant expressions in a programming language. Either a program compiles or it doesn’t. What might happen when you change the code in some unlikely way isn’t immediately relevant to whether the program works now, so it’s easy to skimp on that kind of checking until things get out of hand due to greater scale.
Also, in Starlark, any runtime check you write is a build-time check and calling fail reports a build-time error, which is good enough for users, but not for understanding how to use Starlark functions within Starlark code.
I think it's a cultural thing. People like to think of a language for the build system as a little language that somehow doesn't "deserve" a type system. And even they do think a type system is necessary, they think such a language doesn't "deserve" a complicated type system (say Java-like with subtyping and generics) which makes that type system less useful.
I'm curious, what kind of type system does BUSY use?
The sort of "untyped" that your last sentence is referring to is a dead term, though. The only "untyped" language still in common use is assembler, and that's not commonly written by hand anymore (and when it is, it's primarily running on numbers, not complex structs and complex values). There aren't any extant languages anymore that just accept numbers in RAM and just treat them as whatever.
So increasingly, this objection is meaningless, because nobody is using "untyped" that way anymore. The way in which people do use the term, Python is only "optionally" typed, and a lot of real-world Python code is "untyped".
This "strong typing" message from the Python community has always sounded like propaganda to me - designed to confuse management. Strong typing is about machine checked proofs of invariants, not whether you avoid a few daft built-in coercions.
It depends on what definition of "type system" you're using. Colloquially many programmers use it to refer to any system that checks whether objects have specific shapes. Academics, on the other hand, have a very specific definition of a type system that excludes dynamic detect languages. From TAPL (one of the authoritative works on the subject):
> A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.
And later on:
> A type system can be regarded as calculating a kind of static approximation to the runtime behaviors of the terms in a program. ... Terms like "dynamically typed" are arguably misnomers and should probably be replaced by "dynamically checked," but the usage is standard.
In other words, you're both correct in your definitions depending on who you're talking to, but if we're going to get pedantic (which you seem to be) OP is slightly more correct.
Personally, it feels like dynamically typed language advocates have been getting more and more vocal about their language of choice being "typed" as static typing has grown in popularity in recent years. This seems like misdirected energy—static typing advocates know what they're advocating for and know that dynamically typed languages don't fill their need. You're not accomplishing much by trying to force them to use inclusive language.
Rather than trying to push Python as a typed language it seems like it would be more effective to show why dynamic checks have value.
That's funny: I've been using bazel (and previously blaze) for well over a decade, but it has never once occurred to me to think of starlark as having anything at all to do with Python! I can't see anything about it which is distinctively pythonic.
Your mileage may vary, but Starlark was one of the most complicated languages I've ever read. No types, just layers and layers of indirection, with the goal of making a very complex build rule look "simple". I don't like hiding non-accidental complexity, in particular when the abstraction leaks everywhere. Perhaps it was only due to the Starlark I was exposed to (I worked at Google, probably most of the Starlark code I've read has been written by SREs).
What you describe is really all large-scale Python projects look like. No types, just layers and layers of indirection, with the goal of making the interface simple and Pythonic while hiding implementation complexity. I don't think this is necessarily a fault given that this language explicitly decides to look like Python (and was in fact simplified from Python).
The worst Starlark code I've read has been written not by SREs, but by the Boq team as they have a fetish for accomplishing complicated configuration at build time. This was one of the reasons I've avoided Boq: an incomplete code base that's under development doesn't even begin to build, which is far worse than building something and seeing a real compiler error.
What you describe is really what all programming languages, compilers, interpreters, etc. look like. Layers and layers of indirection, with the goal of making the interface simple while hiding implementation complexity, all the way down to assembly.
How much of this was how bazel works vs. starlark itself?
I find the starlark language is very simple (though inconsistent between the various implementations in bazel, go, and rust) but it takes a bit to understand how the magic between defining rules and implementations works in bazel. and TBH, that is also one place I've really needed auto-completion/static typing in starlark to understand what I can/cannot do.
bazel-lsp gives quite good autocompletion in my experience, for what that's worth. About the only trouble I have still is that sometimes paths in load() calls don't complete properly.
This appears to be the same language that started as a custom build rule DSL for bazel (and evolving from lessons learned in blaze's Python extension), the page positions it as a much more general-purpose embedded language.
I'm all for using the right tool for the job, and I'll acknowledge the benefits of hermeticity and parallelism in many contexts. But, I also think that you need to weigh the cost of adding another language to a project. I don't imagine myself using this anywhere outside of a build system, and even then I would first try to take the declarative rules of that build system as far as I can. It's probably better than shell scripts, though, at least most of the time? May depend on your team's prior familiarity with shell and/or love of Python.
There are three implementations (Go, Rust, Java) which could be a good thing if the language doesn't change often. They would have to be kept in sync for any changes, as well as keep from drifting due to changes in the host language's semantics.
I also think that the closeness in syntax with Python can be a disadvantage, especially if users are expecting more recent Python additions like the walrus operator or pattern matching to be available.
I like to design languages, too, so these comments come from a place of love and understanding.
I think it would help if there were some more specific justification (or examples) of why this is better than something more established (e.g. Lua or MicroPython) or more distinct (e.g. pure data in configs instead of embedded code). I do like that the language attempts to remain as simple as needed.
The syntax is a pure subset but the implementations are bespoke, so I wouldn't equate the established position of Python with whether Starlark is considered established.
I agree that sometimes pure data isn't an option, and I've had to write some Skylark to assist blaze build rules too, but every time it also added tech debt, reduced the number of people on the team who completely understood the build system, and wasn't convenient/practical to test the build extension. The problem I'm referring to above is when the use of a source-controlled data file would have been sufficient but someone had to write a build extension because it's fun or something new to do (or whatever reason seemed convincing at the time).
Then there are people who think that those same data files shouldn't be part of the build process at all and should be part of system turn-up/tear-down, stored in a global config or DB where versioning is alongside the data instead of alongside the program build. Certain kinds of migration are made more difficult if everything is built into the binary. Of course, that strategy comes with its own caveats.
There are so many people at Google just goofing around, lol. People just doing hobby stuff for $350k/year.
There is no justification to implement this custom dialect of Python for a build system that drives everyone crazy 3 times. Reminds me of when it was revealed that 400 people were working on Fuschia — a hobby OS that only shipped on a single smart home device.
One of my hobbies while at Google (2010-2015) was to watch the multiple failed attempts to get rid of Borg Config Language and actual Python in Blaze. It took a lot of work until they eventually succeeded. You're probably underestimating the rigor, the pain, and also the value involved in cleaning those things up: being able to cleanly operate programmatically at scale in the Google monorepo is extremely necessary.
(This is also why I don't trust configuration languages built by people who _didn't_ observe the years of pain. Cue and jsonnet are notable projects that were able to incorporate a lot of lessons.)
IIRC initially blaze was literally using python as a config layer. That made it too easy to write build scripts that were too slow and too hard to optimize and that negatively affected the overall build experience.
An increase in edit/build/run cycle efficiency of thousands of employees justifies an investment of a few engineers.
They could have invented a new bespoke DSL. Instead they choose to stick to a well known and familiar language and just limit its expressiveness to a subset that would be easier to optimize. I think that's quite reasonable
Of the three, only two are Google produced. The rust implementation is written by Facebook for use in their build system. The java implementation is the original but is pretty tied to the the bazel build system and not really very suitable to other uses. The go implementation is meant to be embeddable into a varying number of applications.
Within Google, there a large number of similar tools which are written in go and harness the starlark language. While there are plenty of other options, I will say I think starlark is often a great choice.
I'm not sure. I understand what you are trying to say - do you think we should give up on OS diversity? Are you making fun of a company for actually investing in something ambitious which can benefit many if successful?
While I'm not sure you are in the same crowd, I always think it's interesting that the HN hivemind tends to be upset with the browser monoculture, but doesn't bat an eye at OS monoculture. It feels like you're really just channeling feelings about the company rather than the projects.
Starklark isn't exactly "functional" but by design it is hermetic. You can't call any code that would have a side-effect (other than burning compute resources) or get data that wasn't part of your initial input.
It could make a of sense as a "contract language", or as intended: part of a build system.
I believe Starlark allows side effects the first time it evaluates a file but not subsequent times. Otherwise something as simple as calling .append on a list won't work because that returns None but has an input-mutating side effect. I think it's a good choice.
Starlark (originally Skylark) is the configuration/extension language of Google's build tool Bazel.
It's a bespoke Python subset... functions but no recursion, dicts but no sets, etc.
It's also been adopted by the latest version of Buck (Meta's Bazel analog).
I use it daily.
It's an option for lightweight embeddable scripting language, with implementations in Java and Go. If you want Python familiarity, consider it as an option.
The primitive-recursive property of Cue (https://cuelang.org) is a big draw for me, and may be an alternative worth checking out. The authors have spent a great amount of attention to the type system (they learned a lot of lessons from previous config language designs that did not take lattice theory and unification into account).
Dhall is another configuration language that's deliberately Turing-incomplete. Though its Haskell-inspired syntax turns people off who aren't already Haskell programmers. It's based on calculus of constructions.
I love Starlark. I was a major implementor of it at VGS (the repo is open: https://github.com/verygoodsecurity/starlarky). It had unique distinct features that made it much easier to control and sandbox than many other languages out there.
I even built a codemod library that does a very basic python -> starlark so that one can develop using python ecosystem libraries and just copy & paste into a secure execution environment. It was a huge success at my last company.
I'm very thankful to Laurent Le-brun and Alan Donovan -- both of whom are exceptional engineers that I learned so much from. I thought I was skilled but both of those individuals are just on another level.
Readit News
On one hand, having a "hermetic" subset of Python is nice. You can be sure your Bazel starlark codebase isn't going to be making network calls or reading files or shelling out to subprocesses and all that. The fact that it is hermetic does help make things reproducible and deterministic, and enables paralleization and caching and other things. Everyone already knows Python syntax, and its certainly nicer than the templated-bash-in-templated-yaml files common elsewhere in the build tooling/infra space
On the other hand, a large Starlark codebase is a large Python codebase, and large Python codebases are imperative, untyped, and can get messy even without all the things mentioned above. Even though your Starlark is pure and deterministic, it still easily ends up a rats nest of sphagetti. Starlark goes the extra mile to be non-turing-complete, but that doesn't mean it's performant or easy to understand. And starlark's minimalism is also a curse as it lacks many features that help you manage large Python codebases such as PEP484 type annotations, which also means IDEs also cannot provide much help since they rely on types to understand the code
For https://mill-build.org we went the opposite route: not enforcing purity, but using a language with strong types and a strong functional bent to it. So far it's been working out OK, but it remains to be seen how well it scales to ever larger and more complex build setups
Starlark has been great to build with. You get the readability of having a simple subset of python, without python's dependency management challenges. It is easily extensible with plugin APIs. Concurrent API performance is great without the python async challenges.
One challenge wrt using Starlark as an general purpose embedded scripting language is that it does not support usual error handling features. There are no exceptions and no multi value return for error values, all errors result in an abort. This works for a config language, where a fail-fast behavior is good. But for a general purpose script, you need more fine grained error handling. Since I am using Starlark for API logic only, I came up with a user definable error handling behavior. This uses thread locals to keep track of error state [2], which might not work for more general purpose scripting use cases.
[1] https://github.com/claceio/clace
[2] https://clace.io/docs/plugins/overview/#automatic-error-hand...
A lot of the complaints about Starlark as a programming language, and the proposed alternatives, seem to me to miss out on the UX advantages of having pythonic scripting (which so many folks who have taken a random "coding" class understand intuitively) whereas, e.g., using a lisp or lua would not. Further, having a language and runtime designed for safe use is absolutely critical, and trying to embed another runtime (js/wasm) and manage to lock it down successfully, is a much larger undertaking than I think folks realize.
I wish the starklark team had addressed it at this point.
This brings it to the point. I'm still wondering why the achievements of software engineering of the past fifty years, like modularization and static type checking had apparently so little influence on build systems. I implemented https://github.com/rochus-keller/BUSY for this reason, but it seems to require a greater cultural shift than most developers are already willing to make.
Gradle has its problems, and I often curse it for various reasons, but I'm pretty glad it uses regular languages that I can reuse in non-build system contexts. And the fact that it just bites the bullet and treats build systems as special programs with all the same support that it gives to the programs it's building does have its advantages, even if the results can get quite complex.
Also, in Starlark, any runtime check you write is a build-time check and calling fail reports a build-time error, which is good enough for users, but not for understanding how to use Starlark functions within Starlark code.
I'm curious, what kind of type system does BUSY use?
https://github.com/facebook/buck2/tree/main/starlark-rust/vs...
So increasingly, this objection is meaningless, because nobody is using "untyped" that way anymore. The way in which people do use the term, Python is only "optionally" typed, and a lot of real-world Python code is "untyped".
> A type system is a tractable syntactic method for proving the absence of certain program behaviors by classifying phrases according to the kinds of values they compute.
And later on:
> A type system can be regarded as calculating a kind of static approximation to the runtime behaviors of the terms in a program. ... Terms like "dynamically typed" are arguably misnomers and should probably be replaced by "dynamically checked," but the usage is standard.
In other words, you're both correct in your definitions depending on who you're talking to, but if we're going to get pedantic (which you seem to be) OP is slightly more correct.
Personally, it feels like dynamically typed language advocates have been getting more and more vocal about their language of choice being "typed" as static typing has grown in popularity in recent years. This seems like misdirected energy—static typing advocates know what they're advocating for and know that dynamically typed languages don't fill their need. You're not accomplishing much by trying to force them to use inclusive language.
Rather than trying to push Python as a typed language it seems like it would be more effective to show why dynamic checks have value.
That's funny: I've been using bazel (and previously blaze) for well over a decade, but it has never once occurred to me to think of starlark as having anything at all to do with Python! I can't see anything about it which is distinctively pythonic.
The worst Starlark code I've read has been written not by SREs, but by the Boq team as they have a fetish for accomplishing complicated configuration at build time. This was one of the reasons I've avoided Boq: an incomplete code base that's under development doesn't even begin to build, which is far worse than building something and seeing a real compiler error.
It's pretty easy to add types to Python nowadays. I'd consider it bad practice not to do so in a large project.
I find the starlark language is very simple (though inconsistent between the various implementations in bazel, go, and rust) but it takes a bit to understand how the magic between defining rules and implementations works in bazel. and TBH, that is also one place I've really needed auto-completion/static typing in starlark to understand what I can/cannot do.
I'm all for using the right tool for the job, and I'll acknowledge the benefits of hermeticity and parallelism in many contexts. But, I also think that you need to weigh the cost of adding another language to a project. I don't imagine myself using this anywhere outside of a build system, and even then I would first try to take the declarative rules of that build system as far as I can. It's probably better than shell scripts, though, at least most of the time? May depend on your team's prior familiarity with shell and/or love of Python.
There are three implementations (Go, Rust, Java) which could be a good thing if the language doesn't change often. They would have to be kept in sync for any changes, as well as keep from drifting due to changes in the host language's semantics.
I also think that the closeness in syntax with Python can be a disadvantage, especially if users are expecting more recent Python additions like the walrus operator or pattern matching to be available.
I like to design languages, too, so these comments come from a place of love and understanding. I think it would help if there were some more specific justification (or examples) of why this is better than something more established (e.g. Lua or MicroPython) or more distinct (e.g. pure data in configs instead of embedded code). I do like that the language attempts to remain as simple as needed.
Also I don't think pure data is an option. The point is that you want to generate data, which has many benefits (avoid duplication, easier to test).
It seems that Starlark is a good trade-off given the constraints. My main grief is that it doesn't have type.
I agree that sometimes pure data isn't an option, and I've had to write some Skylark to assist blaze build rules too, but every time it also added tech debt, reduced the number of people on the team who completely understood the build system, and wasn't convenient/practical to test the build extension. The problem I'm referring to above is when the use of a source-controlled data file would have been sufficient but someone had to write a build extension because it's fun or something new to do (or whatever reason seemed convincing at the time).
Then there are people who think that those same data files shouldn't be part of the build process at all and should be part of system turn-up/tear-down, stored in a global config or DB where versioning is alongside the data instead of alongside the program build. Certain kinds of migration are made more difficult if everything is built into the binary. Of course, that strategy comes with its own caveats.
There are so many people at Google just goofing around, lol. People just doing hobby stuff for $350k/year.
There is no justification to implement this custom dialect of Python for a build system that drives everyone crazy 3 times. Reminds me of when it was revealed that 400 people were working on Fuschia — a hobby OS that only shipped on a single smart home device.
(This is also why I don't trust configuration languages built by people who _didn't_ observe the years of pain. Cue and jsonnet are notable projects that were able to incorporate a lot of lessons.)
An increase in edit/build/run cycle efficiency of thousands of employees justifies an investment of a few engineers.
They could have invented a new bespoke DSL. Instead they choose to stick to a well known and familiar language and just limit its expressiveness to a subset that would be easier to optimize. I think that's quite reasonable
An example application is https://github.com/shac-project/shac
Within Google, there a large number of similar tools which are written in go and harness the starlark language. While there are plenty of other options, I will say I think starlark is often a great choice.
While I'm not sure you are in the same crowd, I always think it's interesting that the HN hivemind tends to be upset with the browser monoculture, but doesn't bat an eye at OS monoculture. It feels like you're really just channeling feelings about the company rather than the projects.
Deleted Comment
Deleted Comment
It could make a of sense as a "contract language", or as intended: part of a build system.
It's a bespoke Python subset... functions but no recursion, dicts but no sets, etc.
It's also been adopted by the latest version of Buck (Meta's Bazel analog).
I use it daily.
It's an option for lightweight embeddable scripting language, with implementations in Java and Go. If you want Python familiarity, consider it as an option.
I even built a codemod library that does a very basic python -> starlark so that one can develop using python ecosystem libraries and just copy & paste into a secure execution environment. It was a huge success at my last company.
I'm very thankful to Laurent Le-brun and Alan Donovan -- both of whom are exceptional engineers that I learned so much from. I thought I was skilled but both of those individuals are just on another level.
Starlark Language - https://news.ycombinator.com/item?id=40700549 - June 2024 (49 comments)
An Overview of the Starlark Language - https://news.ycombinator.com/item?id=40573689 - June 2024 (49 comments)
(The) Starlark Language - https://news.ycombinator.com/item?id=39457410 - Feb 2024 (1 comment)
RepoKitteh: Github workflow automation using Starlark - https://news.ycombinator.com/item?id=26674781 - April 2021 (7 comments)
Anyway, I guess "see also:" https://news.ycombinator.com/item?id=42370744