tonsky (u/tonsky) - Readit News

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

Joker_vD · 7 months ago

> If that version doesn’t work, you can try another one.

And how will this look like, if your app doesn't have library C mentioned in its dependencies, only libraries A and B? You are prohibited from answering "well, just specify all the transitive dependencies manually" because it's precisely what a lockfile is/does.

tonsky · 7 months ago

Maven's version resolution mechanism determines which version of a dependency to use when multiple versions are specified in a project's dependency tree. Here's how it works:

- Nearest Definition Wins: When multiple versions of the same dependency appear in the dependency tree, the version closest to your project in the tree will be used.

- First Declaration Wins: If two versions of the same dependency are at the same depth in the tree, the first one declared in the POM will be used.

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

freetonik · 7 months ago

In the world of Python-based end-user libraries the pinned (non-ranged) versions result in users being unable to use your library in an environment with other libraries. I’d love to lock my library to numpy 2.3.4, but if the developers of another library pin theirs to 2.3.5 then game over.

For server-side or other completely controlled environments the only good reason to have lock files is if they are actually hashed and thus allow to confirm security audits. Lock files without hashes do not guarantee security (depending on the package registry, of course, but at least in Python world (damn it) the maintainer can re-publish a package with an existing version but different content).

tonsky · 7 months ago

> I’d love to lock my library to numpy 2.3.4, but if the developers of another library pin theirs to 2.3.5 then game over.

Why? Can’t you specify which version to use?

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

palotasb · 7 months ago

The author is perhaps presenting a good argument for languages/runtimes like JavaScript/Node where dependencies may be isolated and conflicting dependencies may coexist in the dependency tree (e.g., "app -> { libpupa 1.2.3 -> liblupa 0.7.8 }, { libxyz 2.0 -> liblupa 2.4.5 }" would be fine), but the proposed dependency resolution algorithm...

> Our dependency resolution algorithm thus is like this:

> 1. Get the top-level dependency versions

> 2. Look up versions of libraries they depend on

> 3. Look up versions of libraries they depend on

...would fail in languages like Python where dependencies are shared, and the steps 2, 3, etc. would result in conflicting versions.

In these languages, there is good reason to define dependencies in a relaxed way (with constraints that exclude known-bad versions; but without pins to any specific known-to-work version and without constraining only to existing known-good versions) at first. This way dependency resolution always involves some sort of constraint solving (with indeterminate results due to the constraints being open-ended), but then for the sake of reproducibility the result of the constraint solving process may be used as a lockfile. In the Python world this is only done in the final application (the final environment running the code, this may be the test suite in for a pure library) and the pins in the lock aren't published for anyone to reuse.

To reiterate, the originally proposed algorithm doesn't work for languages with shared dependencies. Using version constraints and then lockfiles as a two-layer solution is a common and reasonable way of resolving the dependency topic in these languages.

tonsky · 7 months ago

> would fail in languages like Python where dependencies are shared

And yet Java and Maven exist...

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

zokier · 7 months ago

umm wat

> “But Niki, you can regenerate the lockfile and pull in all the new dependencies!”

> Sure. In exactly the same way you can update your top-level dependencies.

how does updating top-level deps help with updating leaf packages? Is the author assuming that whenever a leaf package is updated, every other package in the dep chain gets immediately new release? That is fundamentally impossible considering that the releases would need to happen serially.

tonsky · 7 months ago

I updated the post, see near the bottom

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

nine_k · 7 months ago

The author seems to miss the point of version ranges. Yes, specific versions of dependencies get frozen in the lock file at the moment of building. But the only way to determine these specific versions is to run version resolution across the whole tree. The process finds out which specific versions within the ranges can be chosen to satisfy all the version constraints.

This works with minimal coordination between authors of the dependencies. It becomes a big deal when you have several unrelated dependencies, each transitively requiring that libpupa. The chance they converge on the same exact version is slim. The chance a satisfying version can be found within specified ranges is much higher.

Physical things that are built from many parts have the very same limitation: they need to specify tolerances to account for the differences in production, and would be unable to be assembled otherwise.

tonsky · 7 months ago

Yeah but version ranges are fiction. Some says: we require libpupa 0.2.0+. Sure you can find a version in that range. But what if it doesn’t work? How can you know that your library will work with all the future libpupa releases in advance?

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

egh · 7 months ago

we've all learned about things, not understood them, and thought "wow, these people must be idiots. why would they have made this complicated thing? makes no sense whatsoever. I can't believe these people, idiots, never thought this through like I have."

Most of us, fortunately, don't post these thoughts to the internet for anybody to read.

tonsky · 7 months ago

I worked for 20 years in an ecosystem that didn’t have lockfiles and had reproducible builds before the term was invented, and now you come and tell me that it couldn’t be?

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

andix · 7 months ago

Lockfiles are essential for somewhat reproducible builds.

If a transient dependency (not directly referenced) updates, this might introduce different behavior. if you test a piece of software and fix some bugs, the next build shouldn't contain completely different versions of dependencies. This might introduce new bugs.

tonsky · 7 months ago

> Lockfiles are essential for somewhat reproducible builds.

No they are not. Fully reproducible builds have existed without lockfiles for decades

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

lalaithion · 7 months ago

What if your program depends on library a1.0 and library b1.0, and library a1.0 depends on c2.1 and library b1.0 depends on c2.3? Which one do you install in your executable? Choosing one randomly might break the other library. Installing both _might_ work, unless you need to pass a struct defined in library c from a1.0 to b1.0, in which case a1.0 and b1.0 may expect different memory layouts (even if the public interface for the struct is the exact same between versions).

The reason we have dependency ranges and lockfiles is so that library a1.0 can declare "I need >2.1" and b1.0 can declare "I need >2.3" and when you depend on a1.0 and b1.0, we can do dependency resolution and lock in c2.3 as the dependency for the binary.

tonsky · 7 months ago

One of the versions will be picked up. If that version doesn’t work, you can try another one. The process is exactly the same

tonsky commented on We shouldn't have needed lockfiles tonsky.me/blog/lockfiles/... · Posted by u/tobr

trjordan · 7 months ago

There is absolutely a good reason for version ranges: security updates.

When I, the owner of an application, choose a library (libuseful 2.1.1), I think it's fine that the library author uses other libraries (libinsecure 0.2.0).

But in 3 months, libinsecure is discovered (surprise!) to be insecure. So they release libinsecure 0.2.1, because they're good at semver. The libuseful library authors, meanwhile, are on vacation because it's August.

I would like to update. Turns out libinsecure's vulnerability is kind of a big deal. And with fully hardcoded dependencies, I cannot, without some horrible annoying work like forking/building/repackaging libuseful. I'd much rather libuseful depend on libinsecure 0.2.*, even if libinsecure isn't terribly good at semver.

I would love software to be deterministically built. But as long as we have security bugs, the current state is a reasonable compromise.

tonsky · 7 months ago

It’s totally fine in Maven, no need to rebuild or repackage anything. You just override version of libinsecure in your pom.xml and it uses the version you told it to

tonsky commented on Sync Engines Are the Future instantdb.com/essays/sync... · Posted by u/GarethX

slifin · a year ago

I'm surprised to see Tonsky here

Mostly because I consider the state of the art on this to be Clojure Electric and he presumably is aware of it at least to some degree but does not mention it

tonsky · a year ago

Clojure Electric is different. It’s not really a sync, it’s more of a thin client. It relies of having fast connection to server at all times, and re-fetches everything all the time. They innovation is that they found a really, really ergonomic way to do it