Writing the packaging expressions should not be harder than writing the program, for the average developer. The average developer doesn't know FP, at this point. Nix needs:
1. Approachability for those not indoctrinated in lazy, declarative, functional programming (i.e. Haskell); nope, Nix Pills are not sufficient for the average dev
2. Editor tooling to help guide the writing of expressions (just as anyone learning C# or Rust can use LSP); a better type system for Nix would help here
3. Better documentation for practical things like "Using Nix to manage the dependencies and package a C++ program using Meson", rather than having people piece this together from a bunch of disparate docs
4. A much better CLI UX than `nix-env -qa` and the like (this is ongoing and experimental, but even that broke recently, causing lots of confusion; now it requires `nix --extra-experimental-features nix-command --extra-experimental-features flakes search nixpkgs`)
5. To seriously answer the question: is the Nix language required for the Nix packaging system to exist? Laziness is required, to some degree, but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.
I feel like Nix folks have been focused so long on solving the tough problems of declarative, deterministic packaging that they haven't been able to focus on the UX. I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else. Perhaps like the early days of Git.
I do really hope Nix succeeds in this, though; I've been using it, or it's been using me, for several years. [1] and [2] for more info.
Having used Nix a bunch, the system has been hard to use and learn in exactly the same ways that messy legacy codebases I've worked with have been hard—which makes sense since Nixpkgs is an absolutely massive codebase that's evolved over the years with very distributed development. People latch onto the Nix language and the paradigm because that's the most tangible difference but, everything else held equal, Nix + Nixpkgs would be no easier to work with if everything were in Python. (And, if my limited experience is any guide, would be worse with an ad-hoc DSL pretending to be Python à la Starlark!)
This is a pattern I've seen over and over: if a project tries to do something different, the novel aspect gets blamed for issues that would not be surprising in any other context. When faced with a massive awful codebase in Java, well, sometimes legacy code can be a pain, right? It is what it is. People rarely jump to "OOP is a mistake" or "Java is completely unusable". But that's exactly what happens all the time with, say, functional programming.
Pointing to an uncommon choice as the underlying cause of a problem makes for a nice, pat story; the pervasive, diffuse problems inherent to messy legacy code do not. After all, here's a single thing that, presumably we can actually change! Nix being functional is an objective fact while people can—reasonably or not—disagree on how well-designed and well-written Nixpkgs-the-codebase is. Besides, even if we could agree, what could we do about it? We all know how much momentum legacy code has and how hard it is to make anything beyond local improvements. Being a socially convenient story doesn't mean it's a particularly strong explanation, but it does mean that people will believe it even if it isn't.
Yeah IMHO nix just needs someone like Brian Kernighan to sit down and write the definitive book about it and its internals. Something that clearly explains the practical and theoretical uses of nix and nixpkgs.
Nix pills, the official manual, etc. aren't really there yet--they talk about all kinds of paradigms that might (?) be going away like nix-env and its ilk. It's just extremely painful to track down the current best practice and way to do things in nix--you're bouncing off all kinds of docs and finding multiple conflicting or out of date ways to do things. Someone needs to give folks a slim book about nix flakes and the flakes way of doing things, the nix language, and a tour of nixpkgs, and IMHO there will be a lot more people using it.
I think the “famous” ‘Simple Made Easy’ talk makes this difference you mean quite palpable: simple is an objective term meaning something is not “interconnected” (not complected) with other parts of the code, while easy is a relative term originating from something being close to us, it being familiar.
So to most programmers, imperative code is easy and FP is hard, but on an objective basis both can be simple. In a way FP may be simpler (according to the talk), as e.g. mutable variables complect/mix together values and time.
I agree, even with FP and (some) Haskell experience, customising Nix and debugging what's going on is made hard by Nixpkgs being a complex codebase more than anything. However, it is different than the usual large web app. The problem of packaging software is fundamentally easy to break down into tiny pieces, the package definitions themselves. Arch's PKGBUILDs were very easy to understand for me as a novice, much better than RPM SPECs or whatever Debian has, so I think it's not impossible for Nix-equivalent packages to be similarly simple.
The biggest barrier, as I see it, is not actually Nix or Nixpkgs code, but the way NixOS differs from every other Linux distro. Most of my delving into Nixpkgs has been motivated by getting something working that would be trivial in a traditional distro. I agree the Nix language is a red herring, what's really needed is to make work other than "adding a package to Nixpkgs" easier. Actually usable introspection tools that the GP mentioned would be great.
> Besides, even if we could agree, what could we do about it?
The same thing we do for other paradigms (system architecture, programming concepts, etc.) - find and document common patterns, then find ways to transition to the patterns away from things out of pattern, and lastly introduce automatic refactorings that do fix these things in bulk.
I definitely agree with you on the evolution thing. Homebrew is a similar DSL for installing packages on macOS. It has the problem where things get deprecated from the DSLs (Casks and Formulae) which leads to not being able to run various commands until previously installed software is fixed.
I'd just like to take a moment to reiterate that object-oriented programming is a mistake produced by cargo-culting half a dozen different styles used by competent programmers (including the devolpers of smalltalk and its relatives, from which OOP appropiated its name), and that Java is completely unusable by anyone with a sense of self-respect and a awareness that programming languages can be better than least-common-denominator noe-COBOLs. I'm not familiar with the specific codebase you're refering, but it would probably be at least less massive and less awful if it had been written in a more decent language, like Haskell or C.
> To seriously answer the question: is the Nix language required for the Nix packaging system to exist? Laziness is required, to some degree, but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.
I'd love to hear from someone deeply familiar with Nix and Guix about laziness.
I'm deeply familiar with Nix and I've concluded that lazy semantics is absolutely critical for a configuration language. It lets me refer to other attributes of my configuration from anywhere. For example, I can refer to port number from my whatever service in my firewall. Nix's system of overlays depends on laziness too to provide efficient late-binding familiar from OOP.
I don't need to topologically sort the evaluation of the various inter-dependencies of my configuration. So long as there exists an evaluation order, laziness finds it.
Laziness is compelling enough that I managed to convince the author of Jsonnet <https://jsonnet.org/> of it when he was designing it, and in turn he helped me design what is now known as overlays in Nix.
I don't even understand how Guix manages to work without laziness, though clearly it does somehow. I'm curious as to how that is possible, though I fear I will only ever truly understand by diving into Guix.
As an 18 month Nix packager/user, I continue to be blown away by how powerful laziness is and how expressive the interactions between multiple overlays can be.
> Nix's system of overlays depends on laziness too to provide efficient late-binding familiar from OOP.
As an OOPer, inheritance is one of the things that is generally trotted out as the worst aspect of OO languages. This leads to the mantra "favor composition over inheritance". Much has been written about why and how. But it seems to cause the same pain points that people talk about with respect to laziness in Nix. "Spooky action at a distance" is one of my favorite ways of describing some of these problems.
(I'm not super familiar with Nix, so I could be misreading the points about laziness. Please forgive me if so.)
> I don't need to topologically sort the evaluation of the various inter-dependencies of my configuration. So long as there exists an evaluation order, laziness finds it.
That’s not a property of laziness but of how Nix evaluates configurations. I think people overvalue what laziness brings them because they attribute to it things which could be done equally well with an eager language.
Recursion and merging semantics are also necessary. I use Nix and Jsonnet a lot, but Nix is much more expressive for complex structures, but the tooling being tightly coupled with the package manage make it impossible to adopt for common use cases.
There's lots of criticism about the Nix expression language, but it's way overblown IMO. The core language is small. It's essentially just JSON with functions, very similar to Jsonnet. But with Nix, people make too big of a deal out of implementation details like it being lazy, functional, etc. I personally like Nix as a configuration language because it's suitable for complex configurations despite it being small and simple.
There's a case to be made about improving the state of the Nixpkgs APIs though. Stabilizing the APIs and documenting it more thoroughly would dramatically change how people perceive the complexity of Nix. Experienced Nix users grep the Nixpkgs repo to learn about its APIs, but that's far from ideal for newcomers who won't even know where they should look for information.
Still, despite its reputation, Nix lowers the bar for software packaging. Package definitions tend to be more concise, customizable, and reusable than the competition. Nixpkgs contains many useful utilities for packaging and there's an abundance of real world examples contained in a single repo. I also frequently work with RPM, but Nix is much easier in comparison.
Point 4 is what lost me when I tried using NixOS for a machine I explicitly wanted as reproducible as possible. I needed to get a bunch of stuff working, then freeze/bundle/lock the useland as a documented working state I could rebuild later if anything went wrong.
When documented examples began steering me towards the still experimental nix flakes and using the sorts of with ever more flags required commands, I bailed and built it using salt. I want to like nix but the current state of things was a massive put off.
flakes are anything but experimental at this point, they've been out for many years and I pretty much use them for everything.
you don't have to type those flags if you add them to the configuration file
> I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else.
Yup. I can't even imagine what could I accomplish if I'd have something like a "python library" for nixpkgs. I still use nixos but gave up fiddling with nix for a while, after trying to wrap some stuff I am using/developing.*
* (Yes, I know it's easy. I just need flakes or correct amount of "lib: prev: { self"s. Sadly not for me.)
> they haven't been able to focus on the UX. I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else.
The majority of communication I've seen from Nix maintainers over the last few months has been about exactly this.
The submitted post is about exactly that while Nix powerful, they want more people to be able to use it without having to pay a steep penalty to learn it.
It also doesn't have anything to do with Haskell or it looking somewhat like Haskell. I'm very comfortable in Haskell, and Nix looks awful to me. It's not that it's functional or it's declarative, it's that it's badly designed and not intuitive, making it hard to remember and therefore awkward to code in.
> Writing the packaging expressions should not be harder than writing the program, for the average developer.
Nah. The logical conclusion of this is that Nix should be some Java-like, Python-like, or whatever else qualifies as something that the average developer knows.
Nix needs to be declarative which rules out the average PL paradigm.
> […] , but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.
Does the average developer know Scheme? I think you just played yourself.
Having just done some prototyping with Nix, this list is spot on in enumerating the pain points I hit.
I think most of the docs problems could be solved with the meta-strategy of “deeply understand all the things the Rust ecosystem got right with docs, discovery, and general pedagogical structure, and replicate that”. Actually have a canonical set of docs. Link liberally between packafes and stdlib. Figure out typing so you can get those hints and links in your IDE. Fix the package docs so they actually tell you what functions the packages export, instead of relying on the wiki for instructions. Etc.
Basically, pay a couple excellent docs people for a few years.
I found the language a bit weird but not terrible to learn. But if it’s possible to build some porcelain for common usecases that skips there need to grok the language, that could be quite transformative. I do wonder if the “dev env/packaging” use-case might be simpler than the “run and configure a whole OS” usecase.
> I think most of the docs problems could be solved with the meta-strategy of “deeply understand all the things the Rust ecosystem got right with docs, discovery, and general pedagogical structure, and replicate that”.
I think every project would love having the money to hire someone full time to work on the documentation and evangelism.
Learning to use a lazy functional programming language remains relatively easy - we use to teach the basis in Ocaml in three hours when I was a student - and Git shows people can tolerate awful CLI tool UX.
I think your point 3 is where most of people issues with Nix lie except I wouldn’t limit myself at a lack of practical enough documentation. Nix documentation pretty much just doesn’t exist period.
If you want to use Nix, you will have to go through the packages or the implementation to understand what’s happening. That’s not very welcoming especially when you consider that’s the other Linux distributions which tried to do things differently in the past and could be a source of users for Nix OS, first of them Gentoo, have absolutely stellar documentation.
Nix (the language) is just JSON with variables and arrow functions.
It's the easiest programming language in the world.
(Which is a problem when people start building huge projects like nixpkgs with it. It's not a language fit for programming-in-the-large, just like Javascript isn't.)
What nix needs more than anything is a centralized repository containing recipes for running all the different kinds of programs, as nic files. ie a dockerhub alternative.
With Eelco as a cofounder of DetSys, I feel really excited about where we're going here. I think the world is in many ways primed and ready for Nix, as long as we can help Nix "meet them in the middle."
We're working on making Nix more accessible and producing good and usable, production-ready workflows so teams can just pick it up and go. I'd love to hear what y'all think, to help make sure we're going in the right direction.
I'm looking at your second link, and it's not really that bad if you are familiar with nix. It's a standard pattern for a package with options. Mostly a long enumerated list, not any different from a long enumerated list of the postgres options themselves. The great thing about it is that it's a single DSL, instead of 10,000 DSLs, one for each package you install.
I do think command line options to do what you ask would be nice, but in order to keep the declarative reprudicibility you get with nix, those commands would need to operate on nix config rather than the system itself. They could update the config then update the system based on the config, so it could look like one step even though it is two behind the scenes.
Yeah, dealing with user directory is a pain, and it is a pain because you can't make it completely functional (what about passwords). And it is the only place I found where nix doesn't work as you would expect it to in my 3 years of nixos use.
Does Eelco's departure from Tweag have any impact on Nickel as a possible replacement for nix?
There are lots of fragmented attempts at making nix easier/more immediately valuable/lower barrier to entry. Some of these are personal projects, some are businesses. Some of these just make nix easier, and some attempt to put something in between nix and the user. I'm thinking about things like Cachix, Flox, and divnix (formerly devos). How do you see your work interacting with these?
One of my barriers to adoption at work has been ensuring maintenance continuity. That's always going to be the case for tools that aren't ubiquitous, but I worry with nix that teams without my assistance will revert to other tools they know better, even if I've invested a good bit of time in teaching them enough to keep things going. Do you seek to address that?
There was some work being done to make Nix work on Windows [0] - but it might have fizzled out. I too would really love Nix to work on Windows. I already use it on both macOS and Linux and it’s great.
I might suggest making a typed superset of Nix, basically adding record types and function signatures. That in itself would be a huge boon for tooling and interacting with nixpkgs. nixpkgs describes a huge and varied meta-API for derivations in various build ecosystems, which makes it hard to use because you ultimately need to dive into source code.
We've heard this feedback pretty frequently. We have a few ideas of ways to add some types to Nix in place. I think even simple naive approaches would add a lot of safety and improve the usability of Nix.
However, one of the bigger problems here is the general API design of Nixpkgs. Most of the function interfaces use some named parameters and then use `...` to accept any additional parameters. All of those then get passed down to some other function, and on and on. This makes a mess! It is hard to tell what functions use what parameters, and none of them can restrict their interface.
We could see some significant DX here by creating smaller, more specific interfaces that actually restrict their inputs.
Some community members have looked into this quite a bit, and personally I have come to the conclusion that types don't really give that much benefit to Nix (and I say this as a Rust & Haskell developer).
Nix expressions are almost always[2] short-lived snippets that evaluate to a data structure, not long-running programs where the distinction between "static analysis time" and runtime is extremely relevant.
In fact, deriving the full set of potentially relevant type constraints is only possible at runtime due to how the import system works and the shape of most expressions. If you pick a single file from nixpkgs without context, you can't get much information from it at all (other than a lot of unknown types being passed to things that accept an unknown type) - it's only in the context of an evaluation of a graph node that uses that file that you can infer anything meaningful about it (and even then that information is only relevant in that context).
I think what people actually want is a way to derive more sensible documentation information from the code so that the various builders etc. could have checked documentation pages, as well as tooling for auto-completing members of attribute sets, names of (formals) function arguments and so on.
fwiw, there's an alternative implementation of the Nix language[0] that we (TVL[1]) are slowly open-sourcing at the moment, and we're aiming to design it in such a way that things like an LSP can be implemented on top of it, and to be able to dump out static information from a context that can help with documentation etc.
I installed nix on my Mac but quickly backed out due to the complexity. I assumed the nix store would just be an ordinary directory with a tool for managing it, similar to brew. I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.
What is the reason for all this machinery? I went with the recommended multi-user install, should I have just used the single user mode instead?
This is used for the daemon, so it doesn't run as root and expose your system to Nix build code.
> adds a separate APFS volume
I think this is required because of macOS security restrictions preventing direct modification of the root directory. The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix, and that can't really change per system because it would break caching and reproducibility. The separate volume is added to /etc/synthetic.conf so it can live in /nix.
The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix,
This is true, but in some sense Nix on Apple Silicon was a missed opportunity. It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix. This would have solved the whole dance needed with volumes and synthetic.conf. I know that there are infrastructure issues (like Hydra using a single Nix store), but it would've made the macOS Nix story so much better.
It's a shame that only Apple can make firmlinks, because that would've been another possible solution (/nix could be a firmlink to the actual store location).
The same problem occurs on e.g. Fedora SilverBlue, because you can also not make arbitrary root directories. But at least on Linux, you don't want to throw away more than one decade of a x86_64 binary cache.
Single-user Nix installs are lighter weight, but also have a major compromise: the store is writable by the user. This means it is pretty easy for software you're using to accidentally change the store underneath Nix, and create unpredictable and unreproducible behavior.
I think the multi-user install is a better Nix experience, even if the install process is spookier.
Like is mentioned elsewhere, Docker does similar contortions to install itself. I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?
It's because Nix wants to install into /nix. Once upon a time doing "sudo mkdir /nix" wasn't a problem, but recent macOS releases have made that very hard.
Nix could switch to an alternate location on macOS (e.g. /opt/nix) but that has a lot of downsides for interoperability with other systems.
The install process was simpler before recent OSX updates.
> I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.
Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?
> What is the reason for all this machinery
Enforcing reproducibility basically.
I don't know all but...
The daemon and APFS volume otherwise readonly /nix that only the daemon can write to can't be created.
The group is probably for the daemon to be able to write to /nix.
The path /nix is important because the remote binary cache paths will miss otherwise and you'll compile everything from source.
> I went with the recommended multi-user install, should I have just used the single user mode instead?
I'm guessing it would work in the way you want, but I always opt for the daemon.
> Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?
Fair observation. I installed nix as a prerequisite for DevBox, discussed here https://news.ycombinator.com/item?id=32600821 . I thought DevBox sounded really cool (and still do!), but the Quickstart took frustratingly long, and ended up not working. Faced with the prospect of debugging it, I opted to cut my losses and uninstall it instead.
That said I'm very much open to trying nix again in the future. Also I want to acknowledge how much effort went into getting /nix to work on the Mac; it appears that was a heavy lift indeed.
It's because builds are completely reproducible, and therefore they have to be built in the same place (in `/nix`), and you can't create directories off of `/` on modern Mac setups.
APFS volume is required to have a read-only nix-store at /nix. The daemon is only required in multi-user installation, and you can opt out of it (by using a single-user install). New groups are for letting people submit jobs to builders in multi-user mode.
Note that the multi-user installation is these days the only suggested approach by the Nix team for new MacOS installations - the old instructions for single user installation on MacOS have been removed from their site.
As someone who has been using NixOS for a couple of years now, I really want to say how appreciative I am of everybody for making noticeable improvements to the system on a somewhat regular basis. The nix command keeps on adding great new features like flake templates and bundling as well as just being more user friendly (error messages, actionable hints, etc.) Additionally, tools like nix-ld [1] make nix more usable than ever with software from external sources. Things just keep on getting better for NixOS users!
Despite the reputation, I feel that NixOS or some derivative of it has the power to become the best distribution for non-technical users in the long run. What NixOS has done is effectively built an interface to every component of a modern Linux system, all that needs to be built is a user application to take advantage of it. Of course, there still needs be some improvements in Nix itself for it to blossom into its final form, but I really see a path to greatness here.
I have often thought about creating a simple unified Win2K-esq or BeOS-like X11 WM/DE specifically for NixOS but unfortunately I lack the time/motivation.
Yes, I could imagine a very locked down opinionated distro built on top of Nix that completely abstracts away all of it and makes it work somewhat like Ubuntu with some prepared nix templates/metaprogramming to automatically generate config. Over time more of the Nix functionality could be exposed through UI or more friendly cli tools as best practices and nix itself evolve, allowing the distro to remain friendly to non-technical users while opening up it's true powers to them.
It wouldn't be able to handle arbitrary manual changes to config at the beginning, so either there would always be a protected generated config region of the whole config and the distro would remain relatively standardized, or an alternative evolution would be a low-code user friendly UI that could map to any possible nix config.
Make the documentation up to the modern standards. If I have to open a single random blog or Github repository to find out what to do and piece it together like a puzzle - it is impossible to adapt at organizational scale.
It is also complex enough to require a modern Language Server.
This has been my experience also. I finished the Nix Pills, and got some personal Python/Haskell projects built using nixpkgs by following the nixpkgs language-specific documentation, but anything off the beaten path is going to involve lots of blog posts and clicking through source code. For example in the Haskell world there are so many blog posts that go in different directions from the nixpkgs docs, using flakes, haskell.nix, etc.
We adopted it in an organization of ~100 engineers, and the only way it’s been possible is having a full-time Nix team writing custom Nix functions specific for our environment and projects. That team also does “Nix help desk” work for one-off questions.
Once it works, it does a great job of hermetic builds, easy Docker images, easy to add cross-repo dependencies mixing C++/Python etc. But there are too many rough edges I can’t recommend it in the general case over Dockerfiles, Bazel or language-specific tooling, Cmake, etc. Pick something simpler, ideally whatever is popular for your language.
Knowing functional programming helps but Nix still is a hard language to learn. I have a lot of Haskell experience but the Nix language was very confusing for me. At first I couldn't even tell a variable name from a keyword (it is a convention to call variables "self" and "super" in some contexts). The Nix language has been created for research in build systems and has evolved over a long time, so it has accumulated quite a lot of idiosyncrasies and cruft.
I believe blaming the language is misguided. The problem is that nixpkgs’ constructs are not well documented, or not yet stable. It would be a problem in any language (you can’t expect someone to feel familiar in a new codebase, and nixpkgs is essentially that).
Yeah, the conventions of nixpkgs are a much bigger obstacle than the language syntax. The language lacks static typing. Also, it is a common convention to create attribute sets crafted for logic that has indirect hard-to-follow links to module where the set is defined. This makes it hard to develop tooling like linters or LSP implementations.
Since Eelco (often treated as Nix's BDFL) and Graham (also very prominent in the Nix community) are involved hopefully this group will actually be able to make traction on some of these issues. This is not even close to the first of these posts saying "we're going to help make Nix more user friendly!", but maybe this group has a chance of making changes to the Nix ecosystem since they can just approve it themselves.
I've had concerns about Nix's governance in the past but it sounds like they may be going in the right direction, so I'm excited to see what they're planning.
"But there’s a catch: to make that happen you need to write some Nix, use Nix tools, and probably consult several documentation sources."
You can use bob[1] if you want a build tool which uses Nix to install dependencies in an easy manner: just list the package names for a task and then they will be installed.
I'm looking forward for all the changes in Nix ecosystem and it's a good sign the fact that they also started working on an initiative to improve Nix documentation which was spread all over the places.
I think they only way in which they are similar is the way a user would add dependencies for a project. If with devbox a user populates devbox.json with package names, with bob you add dependencies in list inside bob.yaml. You can find out more in a short guide[0] I wrote.
But bob intends to be more than a dependency manager, it wants to be fully-fledged build system similar to Dagger, Bazel, Earthly etc with features like:
* build pipelines[1]. You can define and manage your build pipelines in a familiar YAML format. You also get caching for your targets and so on.
* isolated environment. We completely clear the environment for a task. I see with devbox that if I do a `printenv` or an `echo PATH` while I'm still in an active devbox shell, I see variables from my local system which can affect reproducibility.
* workspaces: manage multi-repo setups in an easy manner
How does Bob handle network requirements (downloading dependencies during the build)? I think that is one of the biggest pain points for nix packaing. Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.
I wrote a short guide[1] how bob uses Nix, but essentially we still use `nix-build` under the hood to build a list of packages you give under dependencies
> Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.
Are you referring here to packages which are not available in Nixpkgs?
> Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.
The basic way to fix this with nix is:
1) download the thing with a fixed ouput derivation: a derivation who's source has the URL and checksum of the artifact
2) copy the artifact from the nix store where said build tool stores it
Readit News
1. Approachability for those not indoctrinated in lazy, declarative, functional programming (i.e. Haskell); nope, Nix Pills are not sufficient for the average dev
2. Editor tooling to help guide the writing of expressions (just as anyone learning C# or Rust can use LSP); a better type system for Nix would help here
3. Better documentation for practical things like "Using Nix to manage the dependencies and package a C++ program using Meson", rather than having people piece this together from a bunch of disparate docs
4. A much better CLI UX than `nix-env -qa` and the like (this is ongoing and experimental, but even that broke recently, causing lots of confusion; now it requires `nix --extra-experimental-features nix-command --extra-experimental-features flakes search nixpkgs`)
5. To seriously answer the question: is the Nix language required for the Nix packaging system to exist? Laziness is required, to some degree, but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.
I feel like Nix folks have been focused so long on solving the tough problems of declarative, deterministic packaging that they haven't been able to focus on the UX. I also feel like folks for whom Haskell is comfortable may not realize just how absurd it feels to everyone else. Perhaps like the early days of Git.
I do really hope Nix succeeds in this, though; I've been using it, or it's been using me, for several years. [1] and [2] for more info.
[1]: https://blog.jeaye.com/2015/11/24/nixos/
[2]: https://blog.jeaye.com/2017/07/30/nixos-revisited/
This is a pattern I've seen over and over: if a project tries to do something different, the novel aspect gets blamed for issues that would not be surprising in any other context. When faced with a massive awful codebase in Java, well, sometimes legacy code can be a pain, right? It is what it is. People rarely jump to "OOP is a mistake" or "Java is completely unusable". But that's exactly what happens all the time with, say, functional programming.
Pointing to an uncommon choice as the underlying cause of a problem makes for a nice, pat story; the pervasive, diffuse problems inherent to messy legacy code do not. After all, here's a single thing that, presumably we can actually change! Nix being functional is an objective fact while people can—reasonably or not—disagree on how well-designed and well-written Nixpkgs-the-codebase is. Besides, even if we could agree, what could we do about it? We all know how much momentum legacy code has and how hard it is to make anything beyond local improvements. Being a socially convenient story doesn't mean it's a particularly strong explanation, but it does mean that people will believe it even if it isn't.
Nix pills, the official manual, etc. aren't really there yet--they talk about all kinds of paradigms that might (?) be going away like nix-env and its ilk. It's just extremely painful to track down the current best practice and way to do things in nix--you're bouncing off all kinds of docs and finding multiple conflicting or out of date ways to do things. Someone needs to give folks a slim book about nix flakes and the flakes way of doing things, the nix language, and a tour of nixpkgs, and IMHO there will be a lot more people using it.
So to most programmers, imperative code is easy and FP is hard, but on an objective basis both can be simple. In a way FP may be simpler (according to the talk), as e.g. mutable variables complect/mix together values and time.
The biggest barrier, as I see it, is not actually Nix or Nixpkgs code, but the way NixOS differs from every other Linux distro. Most of my delving into Nixpkgs has been motivated by getting something working that would be trivial in a traditional distro. I agree the Nix language is a red herring, what's really needed is to make work other than "adding a package to Nixpkgs" easier. Actually usable introspection tools that the GP mentioned would be great.
The same thing we do for other paradigms (system architecture, programming concepts, etc.) - find and document common patterns, then find ways to transition to the patterns away from things out of pattern, and lastly introduce automatic refactorings that do fix these things in bulk.
I definitely agree with you on the evolution thing. Homebrew is a similar DSL for installing packages on macOS. It has the problem where things get deprecated from the DSLs (Casks and Formulae) which leads to not being able to run various commands until previously installed software is fixed.
I'd just like to take a moment to reiterate that object-oriented programming is a mistake produced by cargo-culting half a dozen different styles used by competent programmers (including the devolpers of smalltalk and its relatives, from which OOP appropiated its name), and that Java is completely unusable by anyone with a sense of self-respect and a awareness that programming languages can be better than least-common-denominator noe-COBOLs. I'm not familiar with the specific codebase you're refering, but it would probably be at least less massive and less awful if it had been written in a more decent language, like Haskell or C.
I'd love to hear from someone deeply familiar with Nix and Guix about laziness.
I'm deeply familiar with Nix and I've concluded that lazy semantics is absolutely critical for a configuration language. It lets me refer to other attributes of my configuration from anywhere. For example, I can refer to port number from my whatever service in my firewall. Nix's system of overlays depends on laziness too to provide efficient late-binding familiar from OOP.
I don't need to topologically sort the evaluation of the various inter-dependencies of my configuration. So long as there exists an evaluation order, laziness finds it.
Laziness is compelling enough that I managed to convince the author of Jsonnet <https://jsonnet.org/> of it when he was designing it, and in turn he helped me design what is now known as overlays in Nix.
I don't even understand how Guix manages to work without laziness, though clearly it does somehow. I'm curious as to how that is possible, though I fear I will only ever truly understand by diving into Guix.
As an OOPer, inheritance is one of the things that is generally trotted out as the worst aspect of OO languages. This leads to the mantra "favor composition over inheritance". Much has been written about why and how. But it seems to cause the same pain points that people talk about with respect to laziness in Nix. "Spooky action at a distance" is one of my favorite ways of describing some of these problems.
(I'm not super familiar with Nix, so I could be misreading the points about laziness. Please forgive me if so.)
That’s not a property of laziness but of how Nix evaluates configurations. I think people overvalue what laziness brings them because they attribute to it things which could be done equally well with an eager language.
There's a case to be made about improving the state of the Nixpkgs APIs though. Stabilizing the APIs and documenting it more thoroughly would dramatically change how people perceive the complexity of Nix. Experienced Nix users grep the Nixpkgs repo to learn about its APIs, but that's far from ideal for newcomers who won't even know where they should look for information.
Still, despite its reputation, Nix lowers the bar for software packaging. Package definitions tend to be more concise, customizable, and reusable than the competition. Nixpkgs contains many useful utilities for packaging and there's an abundance of real world examples contained in a single repo. I also frequently work with RPM, but Nix is much easier in comparison.
When documented examples began steering me towards the still experimental nix flakes and using the sorts of with ever more flags required commands, I bailed and built it using salt. I want to like nix but the current state of things was a massive put off.
There's a push to make them not be experimental anymore since so much of the community including myself is using them.
Yup. I can't even imagine what could I accomplish if I'd have something like a "python library" for nixpkgs. I still use nixos but gave up fiddling with nix for a while, after trying to wrap some stuff I am using/developing.*
* (Yes, I know it's easy. I just need flakes or correct amount of "lib: prev: { self"s. Sadly not for me.)
The majority of communication I've seen from Nix maintainers over the last few months has been about exactly this.
The submitted post is about exactly that while Nix powerful, they want more people to be able to use it without having to pay a steep penalty to learn it.
Nah. The logical conclusion of this is that Nix should be some Java-like, Python-like, or whatever else qualifies as something that the average developer knows.
Nix needs to be declarative which rules out the average PL paradigm.
> […] , but can the next iteration provide an on-ramp which doesn't involve learning a new lang and paradigm? Guix folks sure think so.
Does the average developer know Scheme? I think you just played yourself.
I think most of the docs problems could be solved with the meta-strategy of “deeply understand all the things the Rust ecosystem got right with docs, discovery, and general pedagogical structure, and replicate that”. Actually have a canonical set of docs. Link liberally between packafes and stdlib. Figure out typing so you can get those hints and links in your IDE. Fix the package docs so they actually tell you what functions the packages export, instead of relying on the wiki for instructions. Etc.
Basically, pay a couple excellent docs people for a few years.
I found the language a bit weird but not terrible to learn. But if it’s possible to build some porcelain for common usecases that skips there need to grok the language, that could be quite transformative. I do wonder if the “dev env/packaging” use-case might be simpler than the “run and configure a whole OS” usecase.
I think every project would love having the money to hire someone full time to work on the documentation and evangelism.
Learning to use a lazy functional programming language remains relatively easy - we use to teach the basis in Ocaml in three hours when I was a student - and Git shows people can tolerate awful CLI tool UX.
I think your point 3 is where most of people issues with Nix lie except I wouldn’t limit myself at a lack of practical enough documentation. Nix documentation pretty much just doesn’t exist period.
If you want to use Nix, you will have to go through the packages or the implementation to understand what’s happening. That’s not very welcoming especially when you consider that’s the other Linux distributions which tried to do things differently in the past and could be a source of users for Nix OS, first of them Gentoo, have absolutely stellar documentation.
It's the easiest programming language in the world.
(Which is a problem when people start building huge projects like nixpkgs with it. It's not a language fit for programming-in-the-large, just like Javascript isn't.)
I've only had to add a few lines to my configuration over 10 PCs and laptops.
We're working on making Nix more accessible and producing good and usable, production-ready workflows so teams can just pick it up and go. I'd love to hear what y'all think, to help make sure we're going in the right direction.
Look at this:
https://nixos.wiki/wiki/PostgreSQL
Now, the real "workflow" with nix is look at other setups, or just look at the code:
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/s...
And look at the code is the only reliable way to see what exactly is supported.
My second gripe is the fact is hard to see what nix do. Today I hit this trouble:
https://www.reddit.com/r/NixOS/comments/x46w98/why_new_user_...
The thing is: Nixos not tell what is doing. `verbose` is too much noise.
What I wish now is something like:
I do think command line options to do what you ask would be nice, but in order to keep the declarative reprudicibility you get with nix, those commands would need to operate on nix config rather than the system itself. They could update the config then update the system based on the config, so it could look like one step even though it is two behind the scenes.
> nix change nothing (if my changes somehow don't do anything, like in my issue)
I miss this feedback greatly having used Ansible since 2013 and only this year diving into a single NixOS installation as my intro.
Does Eelco's departure from Tweag have any impact on Nickel as a possible replacement for nix?
There are lots of fragmented attempts at making nix easier/more immediately valuable/lower barrier to entry. Some of these are personal projects, some are businesses. Some of these just make nix easier, and some attempt to put something in between nix and the user. I'm thinking about things like Cachix, Flox, and divnix (formerly devos). How do you see your work interacting with these?
One of my barriers to adoption at work has been ensuring maintenance continuity. That's always going to be the case for tools that aren't ubiquitous, but I worry with nix that teams without my assistance will revert to other tools they know better, even if I've invested a good bit of time in teaching them enough to keep things going. Do you seek to address that?
I am working on a very large project where multiple solutions have been invented to solve problem which are all handled by Nix:
- Bake in the repository the clone of the dependencies.
- Compile the tool chain to create reproducible builds.
- Create "artifact" builds, to avoid large recompilation times for many developers.
- Use various tools to cache and distribute builds.
- Use a bleeding-edge version of some dependencies.
- Use Node, Java (Android), C++, Rust, …
If only Nix were to run natively on Windows, this would check all the boxes.
[0] - https://github.com/nix-windows/nix
This is also due to the fact that I tend to value Graham's posts highly :)
However, one of the bigger problems here is the general API design of Nixpkgs. Most of the function interfaces use some named parameters and then use `...` to accept any additional parameters. All of those then get passed down to some other function, and on and on. This makes a mess! It is hard to tell what functions use what parameters, and none of them can restrict their interface.
We could see some significant DX here by creating smaller, more specific interfaces that actually restrict their inputs.
Nix expressions are almost always[2] short-lived snippets that evaluate to a data structure, not long-running programs where the distinction between "static analysis time" and runtime is extremely relevant.
In fact, deriving the full set of potentially relevant type constraints is only possible at runtime due to how the import system works and the shape of most expressions. If you pick a single file from nixpkgs without context, you can't get much information from it at all (other than a lot of unknown types being passed to things that accept an unknown type) - it's only in the context of an evaluation of a graph node that uses that file that you can infer anything meaningful about it (and even then that information is only relevant in that context).
I think what people actually want is a way to derive more sensible documentation information from the code so that the various builders etc. could have checked documentation pages, as well as tooling for auto-completing members of attribute sets, names of (formals) function arguments and so on.
fwiw, there's an alternative implementation of the Nix language[0] that we (TVL[1]) are slowly open-sourcing at the moment, and we're aiming to design it in such a way that things like an LSP can be implemented on top of it, and to be able to dump out static information from a context that can help with documentation etc.
[0]: https://cs.tvl.fyi/depot/-/tree/tvix/eval
[1]: https://tvl.fyi/
[2]: Yes, I'm aware that Nix is turing-complete, but surely nobody would write a web framework[3] and HTML templater[4] in it.
[3]: https://cs.tvl.fyi/depot/-/blob/web/bubblegum/README.md
[4]: https://cl.tvl.fyi/c/depot/+/3410
What is the reason for all this machinery? I went with the recommended multi-user install, should I have just used the single user mode instead?
> new Unix group
This is used for the daemon, so it doesn't run as root and expose your system to Nix build code.
> adds a separate APFS volume
I think this is required because of macOS security restrictions preventing direct modification of the root directory. The Nix store has to be housed in /nix because all references to runtime dependencies in the store are absolute paths in /nix, and that can't really change per system because it would break caching and reproducibility. The separate volume is added to /etc/synthetic.conf so it can live in /nix.
> installs a daemon
For multi-user installs, this allows non-privileged users to add build outputs to the Nix store, which ultimately allows these users to share build outputs. More info here: https://nixos.org/manual/nix/stable/installation/multi-user....
This is true, but in some sense Nix on Apple Silicon was a missed opportunity. It started as a blank slate (fresh binary cache) and it would've been a good opportunity to move the store to a writable path like /opt/nix. This would have solved the whole dance needed with volumes and synthetic.conf. I know that there are infrastructure issues (like Hydra using a single Nix store), but it would've made the macOS Nix story so much better.
It's a shame that only Apple can make firmlinks, because that would've been another possible solution (/nix could be a firmlink to the actual store location).
The same problem occurs on e.g. Fedora SilverBlue, because you can also not make arbitrary root directories. But at least on Linux, you don't want to throw away more than one decade of a x86_64 binary cache.
Deleted Comment
I think the multi-user install is a better Nix experience, even if the install process is spookier.
Like is mentioned elsewhere, Docker does similar contortions to install itself. I wonder if it would have been more palatable if the Nix installer was less forward about what it is doing?
I bet most wouldn't consider it spooky if the installer didn't print it out.
If docker started printing it, I bet lots of people would complain it seems similarly complex.
Nix wants to use /nix, but Apple wants macOS to be locked down and secure.
The compromise was putting /nix on a separate volume.
> I discovered it creates a new Unix group, ... installs a daemon
This is at about the same level as Docker, fwiw.
Why does a separate directory need a whole other volume? These are just files, right? Why do they need to be on a separate volume?
Nix could switch to an alternate location on macOS (e.g. /opt/nix) but that has a lot of downsides for interoperability with other systems.
> I discovered it creates a new Unix group, adds a separate APFS volume, installs a daemon. This was too invasive for a tool I was unsure if I even wanted to use, so I uninstalled it.
Since you'd already installed it, wouldn't trying it in some capacity before uninstalling it have made sense?
> What is the reason for all this machinery
Enforcing reproducibility basically.
I don't know all but...
The daemon and APFS volume otherwise readonly /nix that only the daemon can write to can't be created.
The group is probably for the daemon to be able to write to /nix.
The path /nix is important because the remote binary cache paths will miss otherwise and you'll compile everything from source.
> I went with the recommended multi-user install, should I have just used the single user mode instead?
I'm guessing it would work in the way you want, but I always opt for the daemon.
Fair observation. I installed nix as a prerequisite for DevBox, discussed here https://news.ycombinator.com/item?id=32600821 . I thought DevBox sounded really cool (and still do!), but the Quickstart took frustratingly long, and ended up not working. Faced with the prospect of debugging it, I opted to cut my losses and uninstall it instead.
That said I'm very much open to trying nix again in the future. Also I want to acknowledge how much effort went into getting /nix to work on the Mac; it appears that was a heavy lift indeed.
Dead Comment
Despite the reputation, I feel that NixOS or some derivative of it has the power to become the best distribution for non-technical users in the long run. What NixOS has done is effectively built an interface to every component of a modern Linux system, all that needs to be built is a user application to take advantage of it. Of course, there still needs be some improvements in Nix itself for it to blossom into its final form, but I really see a path to greatness here.
I have often thought about creating a simple unified Win2K-esq or BeOS-like X11 WM/DE specifically for NixOS but unfortunately I lack the time/motivation.
[1] https://github.com/Mic92/nix-ld
It wouldn't be able to handle arbitrary manual changes to config at the beginning, so either there would always be a protected generated config region of the whole config and the distro would remain relatively standardized, or an alternative evolution would be a low-code user friendly UI that could map to any possible nix config.
It is also complex enough to require a modern Language Server.
We adopted it in an organization of ~100 engineers, and the only way it’s been possible is having a full-time Nix team writing custom Nix functions specific for our environment and projects. That team also does “Nix help desk” work for one-off questions.
Once it works, it does a great job of hermetic builds, easy Docker images, easy to add cross-repo dependencies mixing C++/Python etc. But there are too many rough edges I can’t recommend it in the general case over Dockerfiles, Bazel or language-specific tooling, Cmake, etc. Pick something simpler, ideally whatever is popular for your language.
Knowing functional programming helps but Nix still is a hard language to learn. I have a lot of Haskell experience but the Nix language was very confusing for me. At first I couldn't even tell a variable name from a keyword (it is a convention to call variables "self" and "super" in some contexts). The Nix language has been created for research in build systems and has evolved over a long time, so it has accumulated quite a lot of idiosyncrasies and cruft.
I hope that a simpler typed language like https://github.com/tweag/nickel replaces it.
I've had concerns about Nix's governance in the past but it sounds like they may be going in the right direction, so I'm excited to see what they're planning.
You can use bob[1] if you want a build tool which uses Nix to install dependencies in an easy manner: just list the package names for a task and then they will be installed.
I'm looking forward for all the changes in Nix ecosystem and it's a good sign the fact that they also started working on an initiative to improve Nix documentation which was spread all over the places.
[1] https://bob.build/
Suggestion: make Bob installable with Nix!
Already created an issue :)
https://github.com/benchkram/bob/issues/193
But bob intends to be more than a dependency manager, it wants to be fully-fledged build system similar to Dagger, Bazel, Earthly etc with features like:
* build pipelines[1]. You can define and manage your build pipelines in a familiar YAML format. You also get caching for your targets and so on.
* isolated environment. We completely clear the environment for a task. I see with devbox that if I do a `printenv` or an `echo PATH` while I'm still in an active devbox shell, I see variables from my local system which can affect reproducibility.
* workspaces: manage multi-repo setups in an easy manner
[0] https://bob.build/docs/usage/package-management
[1] https://bob.build/docs/usage/build
> Many projects have build instructions that depend on downloading stuff in a complex manner that cannot be easily abstracted/shaped to what Nix expects.
Are you referring here to packages which are not available in Nixpkgs?
[1] https://bob.build/docs/usage/package-management
The basic way to fix this with nix is:
1) download the thing with a fixed ouput derivation: a derivation who's source has the URL and checksum of the artifact
2) copy the artifact from the nix store where said build tool stores it