Have there been any good efforts into getting rid of the language or providing an alternative? My effort into switching and the biggest complaint I've read is that the idea behind NixOS and the ecosystem is genius but one of the biggest draw backs is writing everything in Nix.
I spend a weekend every so often defining the core of what I want next time I upgrade, but just find it so annoying I'm sure I won't use anything I've written until there's a major change in the ecosystem.
Putting aside the poor typing (the lack of proper typing is a shame, so valid criticism), I actually really like the language - it's genuinely a great DSL for the particular problems it's supposed to handle.
It does take a bit of use for it to click, though. A lot of it has to do not with Nixlang itself but about learning nixpkgs' idioms.
Way too much sugar/"idioms", which makes it hard for someone new to the language to figure out what a given piece of code is actually doing. Confusing use of semicolons for what almost every other language uses commas or newlines (or nothing) for. It's the same feeling as writing bash, and needing to always look up again exactly what the syntax is and where the semicolons go.
The Nix language also doesn't need to exist. They want to write pure lazy declarative derivations - great, you can do that in any existing language. It's a matter of style and APIs. You don't need to spend years developing a brand new language from scratch. Not to mention that many derivations end up calling a Bash script underneath anyway because at some point you actually need to perform an action in the real world. How would a derivation looked like if written in <insert your favorite scripting language> with lazy APIs?
The Nix language is basically what you'd get if you were designing a DSL for declarative configuration of extremely deeply nested trees. Nearly every "feature" in the language is for making that easier. You could probably write a Nix -> JSON compiler and end up with something completely unreadable because the language hides so much of what's actually present in the fully resolved tree.
That's not true. The language needs to be declarative. if you use your favorite lang and develop a DSL around those APIS, that DSL would be declarative.. also the language itself, isn't that complicated.... it's really quite minimal. It has its roots in ML. I honestly think people expect everything to look like javascript or C++ and it's a shame honestly.
Meh... the nix language being as it is makes it a lot easier to write these things with less cruft. Every attempt I've seen at introducing laziness into a language like python, c++, rust, javascript, etc just seems to require a lot more unnecessary keywords and helper functions and cruft.
There is Guix, which replaces the Nix language with Scheme, but which has some limitations related to a smaller user base, e.g. a smaller package collection.
Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
> Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
That seems like a design flaw in Nix, there's no reason the data model should be so tightly coupled to the scripting implementation that you can't reuse packages written in a different language.
The biggest limitation IMO it's that they are HPC-centric, not caring the desktop, which is the was to allow more people discovering a distro. Also the lack of a proper zfs and lvm/mdraid/luks support it's a big showstopper.
I find the nix language to be quite pleasant. There are some syntax quirks and types would be nice, but in general the “json with functions” vibe is imo great and a very nice fit for the domain. Lots of other modern config languages (e.g. dhall, jsonnet) have ended up in this part of the design space too.
With that said tweag has been working on a kind of nix 2.0 / nix with types for a while with the aim (I think) of being able to use it in nixpkgs: https://github.com/tweag/nickel
I also quite like nixlang for config tasks - in theory! In practice its really annoying. I think the main problem is the interpreter and the bad error messages / bad debuggability.
Part of that just comes from lazy evaluation, which makes debugging a lot harder in general (you feel this in Haskell...), but also just from nix not being a big popular language that gets lots of polish, and being completely dynamically typed.
Nix is ok. I like jsonnet more, and once I've tried to write a converter from jsonnet to nix, but it turned out this is much harder than expected (some idioms don't transfer from nix to jsonnet well.
Me as well. As a haskell / ML programmer, it is extremely intuitive. It's non-innovative (In a good way). Literally it's just a functional scripting language.
It seems inevitable to me that some of the design choices around immutability and isolation are going to result in a larger server image (both on disk and in memory) than if you are prepared to forgo those things. For most people that tradeoff is probably worth it but if you want something to run in an embedded server or with a very low disk footprint it's probably not right for you.
Around 20 years ago people who wanted to do this[2] used to make tiny immutable redhat servers by remounting /usr and a few other things read-only after boot so it's certainly doable but it's a lot more of a pain than what nix does and there is no process isolation and no rollback etc when things go wrong.
[1] ...or generally in fact but that's a matter of opinion and I know people feel differently about this.
It seems to be an issue with testing and debugging, rather than the language itself. The same issue would also be present if you could switch to any other language for configuration.
Nickel lang is such an effort. Id say the syntax is a mix of json and lua and aims for a non-touring complete program. It is still a bit early but it looks promising
No, Nickel is Turing-complete. That's been one of the characteristics intended to distinguish it from most other configuration languages from the start.
yes, https://github.com/garnix-io/garn
But weirdly there was little interest and they rebranded themselves around being a more general build system based on nix instead of what they originally said about being a nix reimplmentation in typescript.
I just finished moving all of my servers off NixOS and I'm finally breathing a sigh of relief.
Deterministic systems are a cool idea, but we're just not there yet. The headaches and pain involved in maintaining these systems and warping the software to obey are too great.
Everything in NixOS works, until it doesn't. And when it doesn't, woe be unto you.
As an opposing viewpoint, I migrated all my self hosted systems (physical machines in a data center) to use nixos and have truly reached self-hosting nirvana. Nothing breaks because I build everything before deployment, and when things do break during system build, it's fairly trivial to fix. I've successfully run all my own infrastructure on NixOS for about 5 years now. No downtime. Very easy to replace computers
What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.
I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
Same experience with Debian. Preaching religious, unquestionable stability but a soon as you install an out of tree software, everything falls apart in your hands.
Any Debian system I used fell apart on upgrades in some unrecoverable way and it was a giant pain in the ass.
The last NixOS stable release I deployed in ~3 hours onto ~30 VMs with only minor issues and everything continued to just work.
I don't actually care about uncommon software stability on Debian. The hypervisor is Debian, so it runs very basic happy path software that always gets tested by the Debian team (and/or Proxmox team) for upgrade problems, and then all of my crazy-ass software runs in containers/VMs which are simple to destroy/rebuild if necessary.
The Proxmox team ensures that the hypervisor stuff works, and now I don't have to worry about a basic change to a nix file resulting in "Unknown entity flibblefrazzle" coming from some random place 18 levels deep in the bowels of the package system/os. It really got to the point where I was afraid to touch anything anymore.
Now I can run an up-to-date Plex! And Chrome Remote Desktop! Without spending 3 whole weekends knee deep in nix guts!
The OP’s article mostly seemed to be about disk space size, specifically when creating a bunch of micro vms. As a NixOS personal dev user, I’m curious about your insights here. Do you have other issues besides the disk space one? I’ve always wanted to do a NixOS server but I only know enough nix to get by on a dev machine. I’m worried about the complexity in deploying a live server.
NixOS is EXTREMELY opaque and obtuse. The documentation also, since everything assumes that you already know everything.
I thought that I could just power through it, but there's just no end to the edge cases, bizarre magic, lack of useful error feedback (if any at all - the most common result is: nothing happens), and situations where you simply cannot do what you want to do.
So you either conform to their model and live with the limitations (and spend countless weekends debugging your builds), or you give up and move on.
I just got tired it all. I want to spend my time USING the computer, not setting it up. So now I use Debian, because everything includes a build for Debian and drivers for Debian. And the best part is: SOMEONE ELSE is maintaining it and keeping it current with the latest security fixes.
> I was trying to bring NixOS to a bare minimum, which is an exercise similar to building containers with the bare minimum required for the software in the container to run. I think this is a worthy endeavour. I think we have all the tools in regular non-docker, non-kubernetes linux to get to a similar outcome, except we won’t need docker or kubernetes or whatever in this new land, thus removing quite a bunch of complexity from the systems we build.
> But doing it on top of NixOS currently feels like a bad path to take.
The author of this blog post might be interested in playing with not-os, another, much smaller OS built with Nix: https://github.com/cleverca22/not-os
Thanks! I have to admit that I've had the itch to build my own NixOS-inspired system more than once, and I haven't done that because I just don't have time to dedicate to this among all the other projects I'm working on. I wasn't aware of not-os before, but I'll definitely dig into the code!
I don't know any other OS where you could even go on such a trip so easily! Figuring out why things are where and being able to disable them like this is pretty cool.
I was really surprised you were able to replace systemdMinimal with systemd in dbus though.
I thought it was there to break the cyclic dependency between systemd and dbus
Depends on what you consider "easy"! Nix itself has a pretty high barrier to entry.
Personally I believe systems that start simple (e.g. Alpine) are easier to mess with. Plus you don't have to give up all benefits of declarative configuration; for example apk has a single file (/etc/apk/world) that defines the exact package set that needs to remain installed. You can edit it and run "apk fix", much like you can edit /etc/nixos/configuration.nix and rerun "nixos-rebuild switch". It's not as powerful as Nixos, but power (and complexity) always has a price.
He didn't mention that nix uses a lot of ram. If your server is tiny and doesn't have swap enabled, running nix command s will make it unresponsive. Are there any nixos alternative that allow you to do system wide configuration from a single source similar to configuration.nix?
As others said, I've moved away from doing nix builds on servers and into a less wasteful (if you're running multiple servers) approach of building once, deploying the bits into a cache, and making the servers fetch them from a cache. I've been slowly working on my own tool to make this workflow easier (nixless-agent, you can find an early version on GitHub).
Do you have a post on your setup? I'm relatively new to NixOS server management and I've been leaning on NixOps for the coordination bits. I'd love to see what you're doing as it sounds more elegant, efficient, and supported.
If you have a different machine with more ram and compute, you can use 'nixos-rebuild --target-host=<server> switch'. That does all the nix building on the local machine, and then just copies binaries and activates the built configuration on the remote machine.
You need not perform your nix evaluation on the same device you are targeting. You can nix copy a system closure to your target and activate it, and there are a number of tools in the nix ecosystem to make this easier.
Granted, if you local machine is low on RAM, or isn't Linux, then you will be in trouble.
The best thing about NixOS is that it's extremely easy to replace systems. On my infrastructure, I have a centralized server where I store all my nix artifacts in /nix/store.
Then on boot for my other systems, they boot into a minimalist nixos image (via netboot) that (1) lookups up the hostname assigned to the system, (2) uses `nix copy` to move the closure of the current system from the main host where I store my builds to the local one, (3) switches into the system (explained below), and (4) then uses nixos's kexec support to switch into the proper kernel booting into the new system.
How to switch into a system: every nixos closure has a top level `bin/switch-to-configuration` script that nixos-rebuild suse. Just run `/nix/store/my-hash/bin/switch-to-configuration switch` and your system will silently be replaced by the new NixOS configuration. Very easy!
I haven't tried it personnally but Guix is similar with config in guile scheme. Have a look at the documentation [1][2].
Caveat: it is a gnu project so no proprietary stuff like firmwares and drivers included out of the box (but there is a community guix nonfree project available [3]). I believe that isn't a problem for virtual machine servers anyway.
Not similar to nix, but you can look into Yocto. You can use it to generate an OS. It is, much more involved than using nix, but suitable for low memory environments.
I agree. The title made me expect a low-quality, clickbait article. The actual article provides a lot of educational value to users of Nix and those interested in trying to build the smallest Linux distribution that can run Nix. Despite using Nix (the package manager) for so long, I haven't considered what it would take to get this. The article answers the question perfectly.
Nix tries to be too many things with too little resources.
As a language it's plagued by very sparse documentation.
As an package manager, it still has the same issues with documentation, (compounded by tools providing the same functionality superseding each other) and then it requires maintaining EVERY software package ever created.
I think the base idea is great, the language is a little weird (but manageable), the DX is unintuitive and badly documented.
The killer reason that made me abandon NixOs is packages not being up to date. I can't fix that and I don't want to spend my life writing my own derivations when the alternative is to use Arch repos + AUR for exotic stuff.
- didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
- instead of trying to strip down the system, maybe go the other way around: only include the command you need with its closure?
closure computation is done in a few places (apparmor profile or systemd.confinement come to mind) and it should be possible to just copy whatever your server binary needs, your kernel (since microVM and not container), and run the binary directly as init (maybe with a simple wrapper that hardcodes network or whatsnot)
> - didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
It does. If you have coreutils from Nixpkgs installed you can check this with
basename "$(realpath "$(which ls)")"
If you see 'ls', you're looking at separate binaries. If you see 'coreutils', it's a symlink to a singular 'coreutils' binary.
I'm not sure about just the one closure since the result needs to boot, but more generally that looks reasonable if your use case fits. There's a couple efforts, IIRC, mostly centered around router firmware where "real" nixos doesn't fit, ex. https://gti.telent.net/dan/liminix
2 is very interesting to me. I'd love to see an example of a "scratch" vm build, in a similar spirit to scratch docker builds. Assume I have all my software and kernel built, what is the minimal filesystem that will boot? How much easier can it get when I am only targeting a VM?
Readit News
I spend a weekend every so often defining the core of what I want next time I upgrade, but just find it so annoying I'm sure I won't use anything I've written until there's a major change in the ecosystem.
Putting aside the poor typing (the lack of proper typing is a shame, so valid criticism), I actually really like the language - it's genuinely a great DSL for the particular problems it's supposed to handle.
It does take a bit of use for it to click, though. A lot of it has to do not with Nixlang itself but about learning nixpkgs' idioms.
I think a good IDE integration could solve this, but not sure how much is possible.
Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
That seems like a design flaw in Nix, there's no reason the data model should be so tightly coupled to the scripting implementation that you can't reuse packages written in a different language.
If nothing changed, they also have a strong ideological drive and funny support any non-free software.
With that said tweag has been working on a kind of nix 2.0 / nix with types for a while with the aim (I think) of being able to use it in nixpkgs: https://github.com/tweag/nickel
Part of that just comes from lazy evaluation, which makes debugging a lot harder in general (you feel this in Haskell...), but also just from nix not being a big popular language that gets lots of polish, and being completely dynamically typed.
It seems inevitable to me that some of the design choices around immutability and isolation are going to result in a larger server image (both on disk and in memory) than if you are prepared to forgo those things. For most people that tradeoff is probably worth it but if you want something to run in an embedded server or with a very low disk footprint it's probably not right for you.
Around 20 years ago people who wanted to do this[2] used to make tiny immutable redhat servers by remounting /usr and a few other things read-only after boot so it's certainly doable but it's a lot more of a pain than what nix does and there is no process isolation and no rollback etc when things go wrong.
[1] ...or generally in fact but that's a matter of opinion and I know people feel differently about this.
[2] me for one, but others also.
Guix is conceptually similar to Nix but uses scheme.
See the 'RATIONALE' document: https://github.com/tweag/nickel/blob/378ece30b3e3c0ab488f659...
The language won't go away and you should try to look at it for more than I don't like it.
The problem isn't the language, the problem is that nixpkgs (and NixOS) are just huge.
Deterministic systems are a cool idea, but we're just not there yet. The headaches and pain involved in maintaining these systems and warping the software to obey are too great.
Everything in NixOS works, until it doesn't. And when it doesn't, woe be unto you.
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
The last NixOS stable release I deployed in ~3 hours onto ~30 VMs with only minor issues and everything continued to just work.
The Proxmox team ensures that the hypervisor stuff works, and now I don't have to worry about a basic change to a nix file resulting in "Unknown entity flibblefrazzle" coming from some random place 18 levels deep in the bowels of the package system/os. It really got to the point where I was afraid to touch anything anymore.
Now I can run an up-to-date Plex! And Chrome Remote Desktop! Without spending 3 whole weekends knee deep in nix guts!
I thought that I could just power through it, but there's just no end to the edge cases, bizarre magic, lack of useful error feedback (if any at all - the most common result is: nothing happens), and situations where you simply cannot do what you want to do.
So you either conform to their model and live with the limitations (and spend countless weekends debugging your builds), or you give up and move on.
I just got tired it all. I want to spend my time USING the computer, not setting it up. So now I use Debian, because everything includes a build for Debian and drivers for Debian. And the best part is: SOMEONE ELSE is maintaining it and keeping it current with the latest security fixes.
Proxmox has its own issues (they all do), but it's a much more inviting experience and nicer community.
> But doing it on top of NixOS currently feels like a bad path to take.
The author of this blog post might be interested in playing with not-os, another, much smaller OS built with Nix: https://github.com/cleverca22/not-os
Could be a decent source of inspiration!
Thanks! I have to admit that I've had the itch to build my own NixOS-inspired system more than once, and I haven't done that because I just don't have time to dedicate to this among all the other projects I'm working on. I wasn't aware of not-os before, but I'll definitely dig into the code!
I was really surprised you were able to replace systemdMinimal with systemd in dbus though.
I thought it was there to break the cyclic dependency between systemd and dbus
Personally I believe systems that start simple (e.g. Alpine) are easier to mess with. Plus you don't have to give up all benefits of declarative configuration; for example apk has a single file (/etc/apk/world) that defines the exact package set that needs to remain installed. You can edit it and run "apk fix", much like you can edit /etc/nixos/configuration.nix and rerun "nixos-rebuild switch". It's not as powerful as Nixos, but power (and complexity) always has a price.
As others said, I've moved away from doing nix builds on servers and into a less wasteful (if you're running multiple servers) approach of building once, deploying the bits into a cache, and making the servers fetch them from a cache. I've been slowly working on my own tool to make this workflow easier (nixless-agent, you can find an early version on GitHub).
Granted, if you local machine is low on RAM, or isn't Linux, then you will be in trouble.
Then on boot for my other systems, they boot into a minimalist nixos image (via netboot) that (1) lookups up the hostname assigned to the system, (2) uses `nix copy` to move the closure of the current system from the main host where I store my builds to the local one, (3) switches into the system (explained below), and (4) then uses nixos's kexec support to switch into the proper kernel booting into the new system.
How to switch into a system: every nixos closure has a top level `bin/switch-to-configuration` script that nixos-rebuild suse. Just run `/nix/store/my-hash/bin/switch-to-configuration switch` and your system will silently be replaced by the new NixOS configuration. Very easy!
Caveat: it is a gnu project so no proprietary stuff like firmwares and drivers included out of the box (but there is a community guix nonfree project available [3]). I believe that isn't a problem for virtual machine servers anyway.
[1] guix cookbook: https://guix.gnu.org/cookbook/en/html_node/index.html#Top
[3] guix manual: https://guix.gnu.org/manual/en/html_node/index.html#Top
[3] https://github.com/guix-users/guix-nonfree
Like you would with Yocto, I just build my systems on a proper host then remotely deploy them.
Also it’s very famous and loved in embedded software circles.
author does not activate new config on host machine, but deploys new host machines as needed.
he evals on build/dev/ci machine only.
As a language it's plagued by very sparse documentation. As an package manager, it still has the same issues with documentation, (compounded by tools providing the same functionality superseding each other) and then it requires maintaining EVERY software package ever created.
I think the base idea is great, the language is a little weird (but manageable), the DX is unintuitive and badly documented.
The killer reason that made me abandon NixOs is packages not being up to date. I can't fix that and I don't want to spend my life writing my own derivations when the alternative is to use Arch repos + AUR for exotic stuff.
- didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
- instead of trying to strip down the system, maybe go the other way around: only include the command you need with its closure? closure computation is done in a few places (apparmor profile or systemd.confinement come to mind) and it should be possible to just copy whatever your server binary needs, your kernel (since microVM and not container), and run the binary directly as init (maybe with a simple wrapper that hardcodes network or whatsnot)
good luck!
It does. If you have coreutils from Nixpkgs installed you can check this with
If you see 'ls', you're looking at separate binaries. If you see 'coreutils', it's a symlink to a singular 'coreutils' binary.I'm not sure about just the one closure since the result needs to boot, but more generally that looks reasonable if your use case fits. There's a couple efforts, IIRC, mostly centered around router firmware where "real" nixos doesn't fit, ex. https://gti.telent.net/dan/liminix
It's for running AppFS inside a VM on QEMU. It uses a statically linked Tcl (which AppFS is written in) to bring the system up.