Highly recommend to check this out, the blog/Arnoult does an amazing job in very succinctly breaking down the aspects of SBOMs in a Nix based infra approach. We can go way beyond the current SLSA levels and provide full provenance at the atomic level of the supply chain for when it's needed. And as Arnoult points out, prune when it's not.
There's good work being done on this across the Nix ecosystem and we have also seen a lot of use for it come in through Flox as well!
Readit Newsronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
Yes, there were various attempts to do this in the container ecosystem, but there is a hard limit on layers on Docker images (because there are hard limits on overlay mounts; you don't really need to overlay all the Nix store mounts of course as they have different paths but the code is for teh geenral case). So then there were various ways of bundling sets of packages into layers, but just managing it directly through Nix store is much simpler.
And I'm back in the land of the living. Can't really beat a response from Justin Cormack!
ronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
How does this differ from the tooling that lets you build containers from nix?
Jotting down a few quick thoughts here but we can totally go deep.
This is something Michael Brantley started working on a few months ago to test out how to make it super easy to ease and leverage existing Nix & Flox architecture.
One of the core differences from my quick perspective is that it specifically leverages the unique way that Flox environments are rendered without performing a nix evaluation, making it safe and optimally performant for the k8s node to realize the packages directly on the node, outside of a container.
ronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
ronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
When I worked on an enterprise data analytics platform, a big problem was docker image growth. People were using different python versions, different cuda versions, all kinds of libraries. With Cuda being over a gigabyte, this all explodes.
The solution is to decompose the docker images and make sure that every layer is hash equivalent. So if people update their Cuda version, it result in a change within the Python layers.
But it looks like Flox now simplifies this via Nix. Every Nix package already has a hash and you can combine packages however you would like.
Yes, this hits the nail on the head. We’ve seen the same explosion in image size and rebuild complexity, especially with AI/ML workloads where Python + CUDA + random pip wheels + system libs = image bloat and massive rebuilds.
With the Kubernetes shim, you can run the hash-pinned environments without building or pulling an image at all. It starts the pod with a stub, then activates the exact runtime from a node-local store.
ronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
I used to love both, Kubernetes and Nix. But after a few years of using both I felt like the abstraction levels are a bit too deep.
Sure, it's easy to stand up a mail server in NixOS, or to just use docker/kubernetes to deploy stuff. But after a few years it felt like I don't have a single understanding of the stack. When shit hits the fan, it makes it very difficult to troubleshoot.
I am now back on running my servers on FreeBSD/OpenBSD and jails or VMM respectively. And also dumbing the stack down to just "run it in a jail, but set it up manually".
The only outlier is Immich. For some reason they only officially support the docker images but not a single clear instruction on how to set it up manually. Sure, I could look at the Dockerfiles, but many of the scripts also expect docker to be present.
And now that FreeBSD also has reproducible builds, it took one more stone away from Nix.
Going to sound weird but with both my hats on I super appreciate this perspective. I can only speak to some areas of Nix and Flox obviously and I know folks are looking into doing this to your point a whole lot better. Zooming in way more into solving for us that just want to run and fix it fast when it breaks.
Also, think it's a huge ecosystem win for FreeBSD pushing on reproducibility too. I think we are trending in a direction where this just becomes a critical principle for certain stacks. (also needed when you dive into AI stacks/infra...)
ronef commented on Run Nix Based Environments in Kubernetes flox.dev/kubernetes/... · Posted by u/kelseyhightower
Ron from Flox here, woke up to feed a brand new 3 day old to see this here! On about 3 hours of sleep (over the lat 48 hours) but excited to try and answer some questions! Feel free to also drop any below <3
We did just launch this last week after a good bit of work from the team. Steve wrote up a deeper technical dive here if anyone is interested - https://flox.dev/blog/kubernetes-uncontained-explained-unloc...
+1 to Farid, great write-up! What you’re seeing is the long-standing “deriver” mismatch: fixed-output derivations can change their .drv without changing the output path. Eelco is calling it out as well in the comment below. I believe the idea behind the path forward is there but happy to hear more!
Also. Check out Farid's other posts.
ronef commented on Free applicatives, the handle pattern, and remote systems exploring-better-ways.bel... · Posted by u/_jackdk_
Bellroy is. very cool and deep in tech. We actually did a Nix in the Wild with them a while back! https://flox.dev/nixinthewild/nix-in-the-wild-bellroy/
There's a lot of reasons to use Nix instead of or WITH Homebrew depending on your exact needs.
Where it’s paid off for me (and where I think it actually wins) is when the problem is recreating environments: multiple machines, teammates, CI, nasty native deps, CUDA stacks, etc. At that point you’re choosing where entropy lives: in invisible drift (brew/manual installs) or in a repo you can diff/rollback.
Also, you don’t always need to go full “immutable everything.” Really depends on your needs here. Hybrid tends to be another sane path. In certain situations this can get you 80% of the upside without having to rip it all out. So kinda the "good enough" which I've seen a lot of folks do.
We (Flox) actually worked on this with Kelsey Hightower a while back - https://bsky.app/profile/kelseyhightower.com/post/3ld2rsccls...