There's no lockfile or anything with this approach right? So in a year or two all of these scripts will be broken because people didn't pin their dependencies?
I like it though. It's very convenient.
Readit Newskorijn commented on AWS forms EU-based cloud unit as customers fret theregister.com/2025/06/0... · Posted by u/rntn
I'm the VP / Distinguished Engineer leading the design, delivery, and operations of the European Sovereign Cloud. I'm in Hamburg right now for the AWS Summit tomorrow.
To answer some questions here in one go - for the European Sovereign Cloud, EU laws always apply. The only people with operational control or access (physical, or logical) are EU people in the EU, and decisions about how lawful orders are handled are also made by EU people in the EU. This is one of the biggest pieces of what it means to be a "Sovereign Cloud" and comes directly from the requirements of our customers. Another is that there are no technical dependencies on non-EU infrastructure.
Of course another answer is that for data access it's also great to build systems like KMS, Nitro, Wickr, CMK encryption, etc ... where we as an operator simply have no access to customer data in the first place. And those protections stand too.
Are you an EU citizen?
korijn commented on Bauplan – Git-for-data pipelines on object storage docs.bauplanlabs.com/en/l... · Posted by u/barabbababoon
Plenty of stuff in common with dbt's philosophy. One big thing though, dbt does not run your compute or manage your lake. It orchestrate your code and pushes it down to a runtime (e.g. 90% of the time Snowflake).
This IS a runtime.
You import bauplan, write your functions and run them in straight into the cloud - you don't need anything more. When you want to make a pipeline you chain the functions together, and the system manages the dependencies, the containerization, the runtime, and gives you a git-like abstractions over runs, tables and pipelines.
I see, this is a great answer. So you don't need any platform or spark or anything. Just storage and compute?
korijn commented on Bauplan – Git-for-data pipelines on object storage docs.bauplanlabs.com/en/l... · Posted by u/barabbababoon
Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic.
Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizations at the intersection of FaaS and data - e.g. rebuilding functions can be 15x faster than the corresponding AWS stack (https://arxiv.org/pdf/2410.17465). All capabilities are available to humans (CLI) and machines (SDK) through simple APIs.
Would love to hear the community’s thoughts on moving data engineering workflows closer to software abstractions: tables, functions, branches, CI/CD etc.
How does this compare to dbt? Seems like it can do the same?
korijn commented on Software development topics I've changed my mind on chriskiehl.com/article/th... · Posted by u/belter
> Most won't care about the craft. Cherish the ones that do, meet the rest where they are
> (…)
> People who stress over code style, linting rules, or other minutia remain insane weirdos to me. Focus on more important things.
What you call “stressing over minutiae” others might call “caring for the craft”. Revered artisans are precisely the ones who care for the details. “Stressing” is your value judgement, not necessarily the ground truth.
What you’re essentially saying is “cherish the people who care up to the level I personally and subjectively think is right, and dismiss everyone who cares more as insane weirdos who cannot prioritise”.
Is it ironic that this comment has evoked a discussion on minutiae?
Take ownership.
korijn commented on Python Has Too Many Package Managers dublog.net/blog/so-many-p... · Posted by u/LarsDu88
Try uv. You will be shocked how much faster it is.
I will! I'm sure it's faster when the data is available. But when it's not, in the common circumstances described above, network and disk IO are still the same unchanged bottlenecks, for any package manager.
In conversations like this, we are all too quick to project our experiences on the package managers and not sharing in what circumstances we are using them.
korijn commented on Python Has Too Many Package Managers dublog.net/blog/so-many-p... · Posted by u/LarsDu88
I have worked with poetry professionally for about 5 years now and I am not looking back. It is exceptionally good. Dependency resolution speed is not an issue beyond the first run since all that hard to acquire metadata is actually cached in a local index.
And even that first run is not particularly slow - _unless_ you depend on packages that are not available as wheels, which last I checked is not nearly as common nowadays as it was 10 years ago. However it can still happen: for example, if you are working with python 3.8 and you are using the latest version of some fancy library, they may have already stopped building wheels for that version of python. That means the package manager has to fall back to the sdist, and actually run the build scripts to acquire the metadata.
On top of all this, private package feeds (like the one provided by azure devops) sometimes don't provide a metadata API at all, meaning the package manager has to download every single package just to get the metadata.
The important bit of my little wall of text here though is that this is all true for all the other package managers as well. You can't necessarily attribute slow dependency resolution to a solver being written in C++ or pure python, given all of these other compounding factors which are often overlooked.
korijn commented on Python Gotcha: strip, lstrip, rstrip can remove more than expected andrewwegner.com/python-g... · Posted by u/synergy20
RTFM
https://en.wikipedia.org/wiki/Literacy_in_the_United_States