> but the second is to clearly describe a compelling reason for people to adopt.

Build source straight from Git:

   nix run github:someuser/someproject

Need a different version?

   nix run github:someuser/someproject?ref=v1.0.0

Wanna replace some dependency?

   nix run \
      --override-input somelib github:someotheruser/somelib \
      github:someuser/someproject

Wanna fork:

   git clone https://github.com/someuser/someproject
   # do your changes
   nix run someproject/

And the best part is, it's conceptually very simple, it's mostly just a bunch of symlinks and environment variables behind the scenes. If you wanna inspect what's in a package, just `cd /nix/store/yourpackage-HASH` and look around.

NixOS just feels like a distribution build from the ground up for Free Software. The "reproducibility" in every day use just means that stuff won't randomly break for no reason. And if you don't wanna go the full NixOS route, you can just install the Nix package manager itself on any other distribution.

That said, the part where Nix gets painful is when it has to interact with the rest of the software world. Things like software that wants to auto-update itself really does not fit into the Nix ecosystem at all and can be rather annoying to get to work.

There are of course numerous other pain points, missing features and all that. But being able to flip between versions, fork, compile and all that with feels just so much better than anything else.

I disagree, I think the value proposition for reproducibility is clear, it's just that the learning curve "is too damn high!" I'm highly motivated to learn and use Nix (or Guix for that matter) but I've bounced off of it three or four times now, and I'm the kind of weirdo who learns new PLs for fun.

Someone once said that you don't learn Nix, you reverse engineer it.

Two of the major differences between terraform and Nix that I see are 1. it’s possible to muddle through in terraform and 2. Hashicorp has put a non-trivial amount of effort into documentation for all levels of users. I’ve taken a stab at using Nix for Rust projects and could not even get to a point where I had something that functioned. I found plenty of material online but was it out of date, idiosyncratic, did it use flakes or not, etc etc? I suppose I could have contorted my existing project to meet the examples I found in various GitHub repos but my stuff is bog standard Rust so I don’t know that I’d be willing to. As for documentation, what should I, as a new and invested user, be looking for? Are flakes the future? Are they a distraction? Why are all the suggested docs I could find several year old guides on blogs? There’s 20 years of information floating around and the official project documentation, well, I don’t know who the audience is but it’s not learners.

It’s a shame. The promise of Nix/NixOS is really interesting — being able to deterministically create VMs with a custom user land is desirable to me — but in practice I can’t even get a simplistic project to compile, let alone something elaborate. Terraform is jank but it’s not a whole language that needs to be learned, seemingly, before the official docs start to become coherent in their underlying context.

> We need to accept that "reproducibility" isn't quite working...

I guess it doesn't sell Nix as strongly as it could..

But, it's hardly for a lack of enthusiasm on Nix user's part. -- Rather, I've seen a few "what's nix good for anyway" comments, and this results in many lengthy replies extolling nix.

Agreed, reproducibility is only one aspect of Nix and doesn't quite capture the whole picture. That's why so many newcomers see Nix as nothing more than a Docker replacement. There's also too much misconceptions about Nix the language that's scaring people off.

I'd like to see more being discussed about:

* Its unique ability to treat packages as programmable data (i.e., derivations)

* Its use case as a building block for deployment systems that knows about and integrates with packages

* Its JSON-like simplicity

They're all central to the Nix experience, and yet it's often overlooked in Nix discussions.

I wonder if the corporate backing behind Docker has anything to do with Nix being adopted less. There’s a lot of overlap between Nix and Docker, and Docker had major corporate guns behind it from early on. Reproducibility as you mentioned, is easily achieved with Docker, and there is no need to learn the Nix ecosystem.

Personally, I want to learn Nix, but Ive never forced myself to do it because it’s so much easier to make a Docker image do what I want. Nix is the pinnacle of a reproducible environment that doesn’t randomly break, but Docker is 80% of the way there and much easier.

It’s like comparing trucks (Docker) with planes (Nix) for logistics. I can pay out the wazoo for a plane to get my package there over night, or I can pay a small fraction to wait a few days.

Your email form to follow the book is returning errors to me. And you have no other contact channels I can find. Might want to look into that.

Excited to see this being in progress. Have you checked out the docs effort spun up with the Nix Documentation team?

I think this project is really promising.

My main concern is that it puts another layer of abstraction atop an already complex (and at times leaky) abstraction.

I’d love to see more clear docs about what devenv is actually doing under the covers, and how to escape-hatch into Nix land when I inevitably need to tweak something.

Also, similarly, how do I map Nix docs (often just a set of example expressions) into equivalent devenv incantations?

(It’s been a few months since I last looked so maybe things have come along since then.)

Thanks Domen,

devenv.sh is quite nice, and have already started using it.

DevEnv looks neat, maybe I’ll try it.

I expect my main hurdle will be stuff that isn’t in NicPkgs. Especially for dev stuff, I’m going to want to pull in low-profile GitHub stuff or Python packages. What’s the escape hatch to bring those into the dev environment?

This looks like a really nice UX improvement on top of typical Flake devshells.

Very cool!

A few antipatterns/annoyances I've come across over the years:

Importing paths based on environment variables:

There is built-in support for this, e.g. setting the env var `NIX_PATH` to `a=/foo:b=/bar`, then the Nix expressions `<a>` and `<b>` will evaluate to the paths `/foo` and `/bar`, respectively. By default, the Nix installer sets `NIX_PATH` to contain a copy of the Nixpkgs repo, so expressions can do `import <nixpkgs>` to access definitions from Nixpkgs.

The reason this is bad is that env vars vary between machines, and over time, so we don't actually know what will be imported.

These days I completely avoid this by explicitly un-setting the `NIX_PATH` env var. I only reference relative paths within a project, or else reference other projects via explicit git revisions (e.g. I import Nixpkgs by pointing the `fetchTarball` function at a github archive URL)

Channels:

These always confused me. They're used to update the copy of Nixpkgs that the default `NIX_PATH` points to, and can also be used to manage other "updatable" things. It's all very imperative, so I don't bother (I just alter the specific git revision I'm fetching, e.g. https://hackage.haskell.org/package/update-nix-fetchgit helps to automate such updating).

Nixpkgs depends on $HOME:

The top-level API exposed by the Nixpkgs repository is a function, which can be called with various arguments to set/override things; e.g. when I'm on macOS, it will default to providing macOS packages; I can override that by calling it with `system = "x86_64-linux"`. All well and good.

The problem is that some of its default values will check for files like ~/.nixpkgs/config.nix, ~/.config/nixpkgs/overlays.nix, etc. This causes the same sort of "works on my machine" headaches that Nix was meant to solve. See https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/...

I avoid this by importing Nixpkgs via a wrapper, which defaults to calling Nixpkgs with empty values to avoid its impure defaults; but still allows me to pass along my own explicit overrides if needed.

The imperative nix-env command:

Nix provides a command called 'nix-env' which manages a symlink called ~/.nix/profile. We can run commands to "install packages", "update packages", "remove packages", etc. which work by building different "profiles" (Nix store paths containing symlinks to a bunch of other Nix store paths).

This is bad, since it's imperative and hard to reproduce (e.g. depending on what channels were pointing to when those commands were run, etc.). A much better approach is to write down such a "profile" explicitly, in a git-controlled text file, e.g. using the `pkgs.buildEnv` function; then use nix-env to just manage that single 'meta-package'.

Tools which treat Nix like Apt/Yum/etc.

This isn't something I haven't personally done, but I've seen it happen in a few tools that try to integrate with Nix, and it just cripples their usefulness.

Package managers like Apt have a global database, which maps manually-written "names" to a bunch of metadata (versions, installed or not, names of dependencies, names of conflicting packages, etc.). In that world names are unique and global: if two packages have the name "foo", they are the same package; clashes must be resolved by inventing new names. Such names are also fetchable/realisable: we just plug the name and "version number" (another manually-written name) into a certain pattern, and do a HTTP GET on one of our mirrors.

In Nix, all the above features apply to "store paths", which are not manually written: they contain hashes, like /nix/store/wbkgl57gvwm1qbfjx0ah6kgs4fzz571x-python3-3.9.6, which can be verified against their contents and/or build script (AKA 'derivation'). Store paths are not designed to be managed manually. Instead, the Nix language gives us a rich, composable way to describe the desired file/directory; and those descriptions are evaluated to find their associated store paths.

Nixpkgs provides an attribute set (AKA JSON object) containing tens of thousands of derivations; and often the thing we want can be described as 'the "foo" attribute of Nixpkgs', e.g. '(import <nixpkgs> {}).foo'

Some tooling that builds-on/interacts-with Nix has unfortunately limited itself to only such descriptions; e.g. accepting a list of strings, and looking each one up in the system's default Nixpkgs attribute set (this misunderstanding may come from using the 'nix-env' tool, like 'nix-env -iA firefox'; but nix-env also allows arbitrary Nix expressions too!). That's incredibly limiting, since (a) it doesn't let us dig into the structure inside those attributes (e.g. 'nixpkgs.python3Packages.pylint'); (b) it doesn't let us use the override functions that Nixpkgs provides (e.g. 'nixpkgs.maven.override { jre = nixpkgs.jdk11_headless; }'); (c) it doesn't let us specify anything outside of the 'import <nixpkgs> {}' set (e.g. in my case, I want to avoid NIX_PATH and <nixpkgs> altogether!)

Referencing non-store paths:

The Nix language treats paths and strings in different ways: strings are always passed around verbatim, but certain operations will replace paths by a 'snapshot' copied into the Nix store. For example, say we had this file saved to /home/chriswarbo/default.nix:

  # Define some constants
  with {
    # Import some particular revision of Nixpkgs
    nixpkgs = import (fetchTarball {...}) {};

    # A path value, pointing to /home/chriswarbo/defs.sh
    defs = ./defs.sh;

    # A path value, pointing to /home/chriswarbo/cmd.sh
    cmd = ./cmd.sh;
  };
  # Return a derivation which builds a text file
  nixpkgs.writeScript "my-super-duper-script" ''
    #!${nixpkgs.bash}/bin/bash
    source ${nixpkgs.lib.escapeShellArg defs}
    ${cmd} foo bar baz
  ''

Notice that the resulting script has three values spliced into it via ${...}:

- The script interpreter `nixpkgs.bash`. This is a Nix derivation, so its "output path" will be spliced into the script (e.g. /nix/store/gpbk3inlgs24a7hsgap395yvfb4l37wf-bash-5.1-p16 ). This is fine.

- The path `cmd`. Nix spots that we're splicing a path, so it copies that file into the Nix store, and that store path will be spliced into the script (e.g. /nix/store/2h3airm07gp55rn9qlax4ak35s94rpim-cmd.sh ). This is fine.

- The string `nixpkgs.lib.escapeShellArg defs`, which evaluates to the string `'/home/chriswarbo/defs.sh'`, and that will be spliced into the script. That's bad, since the result contains a reference to my home folder! The reason this happens is that paths can often be used as strings, getting implicitly converted. In this case, the function `nixpkgs.lib.escapeShellArg` transforms strings (see https://nixos.org/manual/nixpkgs/stable/#function-library-li... ), so:

- The path `./defs.sh` is implicitly converted to the string `/home/chriswarbo/defs.sh`, for input to `nixpkgs.lib.escapeShellArg` (NOTE: you can use the function `builtins.toString` to do the same thing explicitly)

- The function `nixpkgs.lib.escapeShellArg` returns the same string, but wrapped in apostrophes (it also adds escaping with backslashes, but our path doesn't need any)

- That return value is spliced as-is into the resulting script

To avoid this, we should instead splice the path into a string before escaping; giving us nested splices like this:

    source ${nixpkgs.lib.escapeShellArg "${defs}"}

For what reason do you "still need Docker?"

With Linux, the only stable base required for Nix to function is the kernel. Nix packages all the required dependencies right down to glibc. Since the Linux kernel famously "doesn't break userspace," any sufficiently new kernel would suffice. Until recently, I've been able to get the latest Nix packages working on an ancient Linux 2.6 kernel. And even the kernel can be managed with Nix if you use NixOS. But Docker can't, so it's no use here.

As for Terraform, I don't see how it's relevant to this discussion. Nix-based SSH deployment tools can replace some of its functionality, so perhaps that's what you're talking about?

This is not true. Nix is supported on several non-Linux platforms, including macOS and Windows WSL.

I'm definitely going to give that a try. I had issues with the Bash script on macOS and was annoyed that it left me with a system that I had to manually clean up.

Why would engineers need their environment shipped to them by a human?

With the Hydra binary cache I can quite comfortably update my Avoton C2550 based router or Raspberry Pi 4 AirPlay receiver within a few minutes. It’s only if I need to build a non trivial package that I have to make sure I build on my desktop or in a VM on my MacBook.

And here I was thinking about Gentoo...

> Just because some package didn't require another package at build time doesn't mean we don't ever want to use them together at runtime. Yet Docker conflates the two, introducing all sorts of unnecessary friction all over the place.

But Docker in fact doesn't have compiletime dependencies, it needs you to specify runtime deps only. If you want to build something in Docker, you use two-stage builds and the runtime deps of the first stage become your compiletime deps.

> This is why something as simple as getting more than one process to work with each other on Docker is such an overcomplicated mess

I don't get this, why is it considered an overcomplicated mess? If you want to run several processes in one container, you just launch it with a lightweight process manager, if you want to run it in separate containers -- well, that's even easier, just launch separate containers and configure the communication between them with a network.

> Nix gets this right with better design and composable packages

Nix actually implements it worse than Docker in some sense. Particularly, the exact problem that you described:

> Just because some package didn't require another package at build time doesn't mean we don't ever want to use them together at runtime

is not solved in Nix, runtime deps must be a subset of compiletime deps.

> that's neither composable nor customizable

Compositionality is a completely different issue on which I do have problems with Docker. DAG-oriented environment building is strictly better than inheritance-oriented, but that's all orthogonal with compiletime-runtime separation.

The choice of topology of the package and layering system - be it a tree as in OCI or a more general graph topology as in Nix - is only a very small part of either of these systems. I agree that the general graph topology is superior in some points.

My point though was that at the implementational level, the old symlink-based way of implementing it in Nix is severely lacking the isolation and more general security capabilities of the bind-mount, namespaces and cgroup-based approach of OCI and other newer packaging systems. And Nix needs to implement that.

My impression is that the OCI package spec isn't what would be in the way of implementing a system that would combine the isolation and security of a system based on bind-mount, namespaces and cgroups like OCI with a graph topology like Nix and that there would thus be an opportunity to combine the two, which would help Nix take over the dominating position in that space. If OCI turned out to be impossible to use with the more graph-based approach of Nix, then that would mean a much higher implementational work - not that it couldn't be done, but it still would need to be done then. But either way, Nix cannot stay with its old symlink-based layering: Failing to implement the security features that are now expected from modern packaging systems (isolation, bind mounts, cgroups, namespaces etc) is a surefire way to progressively maneuver into irrelevance.

Nix has a window of opportunity here due to the current weakness of the big players in the field. But it can't afford to let that slip.