git-annex does support rclone as a special remote iirc
Yes it does, but I don't think the special remote supports the importtree feature, which would be necessary for this.
Readit NewsDoes this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)?
My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?
In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import <branch>:<subdir> --from <sharepoint/dropbox-remote>` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision).
Unfortunately this is not (yet?) supported I think. But you could also just do something like this: `rclone copy/sync <sharepoint/dropbox-remote>: ./<local-directory> && git annex add ./<local-directory> && git commit -m <message>`.
I've used this for years, but to me the big selling point was integration with cloud storage providers as a means of backup. That, however, was always flaky and dependent on unmaintained third-party plugins. I think there was also a bug at some point that caused some data inconsistencies, so eventually I stopped.
Does anyone know if the situation has improved on that front in the past 5 years?
Depends on the cloud storage provider, I think. The best chances are with those that support the more standard protocols like S3, webdav, sftp, etc.. A relatively new development is the special remote built into rclone, which should be better maintained than some other third-party special remotes and provides access to all rclone-supported remotes.
How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so).
I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.
I had tested a git-annex repository with about 1.5M files and it got pretty slow as well. The plain git repo size grew to multiple GiB and plain git operations were super slow, so I think this was mostly a git limitation. DataLad's approach of nested subdatasets (in practice git submodules where each submodule is a git-annex repository) can help, if it fits the data and workflows.
matrss commented on Fuse is 95% cheaper and 10x faster than NFS nilesh-agarwal.com/storag... · Posted by u/agcat
(Posting while the title is "Fuse is 95% cheaper and 10x faster than NFS", I'm guessing that will get changed based on the HN rules)
This is not at all about NFS vs FUSE, this is about specific NFS providers vs specific FUSE with some specific object store backends.
FUSE us just a way to have a filesystem not implemented in the kernel. I can have a FUSE driver that implements storage based on rat trained to push a button in reaction to lights turning on, or basically anything else.
NFS is a specific networked filesystem.
> NFS is a specific networked filesystem.
NFS is a set of protocols for networked filesystems. You can just as well implement an NFS server that "implements storage based on rat trained to push a button in reaction to lights turning on". Some people even argue it is a better way to do it than FUSE, because you get robust clients on most platforms with included caching out of the box. E.g. this is a library for building such a NFS server: https://github.com/xetdata/nfsserve
matrss commented on Anaconda Raises $150M Series C anaconda.com/press/anacon... · Posted by u/diverted247
362 downloads over 20 hours isn't that impressive. Not saying Anaconda isn't great. Don't have much experience with it but I hope the devs get paid for all the value delivered, which I think is largem
Well, it is more than a handful, that was the only point.
You seem to be conflating Anaconda with the conda ecosystem. This package is from conda-forge, which is a community project sponsored by Anaconda but otherwise unrelated to it.
I couldn't care less for Anaconda, but with conda-forge and pixi there is a decent general purpose and language-agnostic package management tool for development purposes in the conda ecosystem.
matrss commented on Anaconda Raises $150M Series C anaconda.com/press/anacon... · Posted by u/diverted247
I admit I'm not super well versed in what conda's main uses are. Python's whole tooling situation has felt like a nightmare & I've tried to keep far away, but I've had to face it a lot more recently because it's so prevalent for AI. Thankfully uv seems to have done a huge amount of what I need.
Still, if the concern is language-agnostic ways to use tooling, mise (nee rtx) is the 1000 pound gorilla in the room today. Incredibly fast well built Rust based tool that has really massively expanded in scope & offerings, with grace & elegance. I thought it was an asdf replacement, for installing/using toolchains, for .tool-versions files. But it's really grown to be a lot more, capable of letting you isolatedly manage tools it can install from a huge variety of backends (pip, npm, cargo, others). https://mise.jdx.dev/dev-tools/backends/
Hadn't heard about that before, looks interesting. But AFAICS mise focuses on installing tools, not actually all dependencies. It integrates with the languages dependency management tools, which doesn't help you if the language doesn't even have one or the integration wasn't written yet.
I'd rather see more adoption of guix for this purpose. It is a single package manager with a functional approach that allows for introspection of all dependencies (down to the bootstrap toolchains used to build the bootstrap toolchains that build your toolchains, which is something that AFAIK no other package manager except for nix can do), has a fairly large package repository, straightforward locking, actually tested packages, and very easy build recipes (unlike conda-forge...).
matrss commented on Anaconda Raises $150M Series C anaconda.com/press/anacon... · Posted by u/diverted247
See, why are you adding forge in there? Why do you think forge exists?
What does it change? Conda-forge users are users of the conda ecosystem. You suggested there is only a handful of those that use it not for python, I provided a counter point.
Or did I misunderstand you and what you meant with "conda" was either anaconda, or conda-the-software? But then the comment about Python doesn't make much sense.
matrss commented on Anaconda Raises $150M Series C anaconda.com/press/anacon... · Posted by u/diverted247
And how many people (count them with fingers) use Conda for anything other than Python? It’s a bloatware. People stopped using Conda because these people kept making the bloat worse.
A single one of the gfortran packages released about 20 hours ago already has 362 downloads, so I think it is safe to say that the non-python users of conda-forge are more than just a handful.
If you don't need it that's fine, no one is stopping you from using PyPI and uv instead. But for some that is not a replacement.
And yes, some things about conda(-forge) could be described as bloated. I particularly dislike the convoluted packaging process.
It is always tradeoffs and deciding based on your own use cases. E.g. if you want to distribute tested packages to users of your software then both conda-forge and PyPI are ill-suited for you. They (and most other package managers) do install-time dependency resolution, so the installation cannot be guaranteed to be tested as working at all. Some package managers do that better, so is conda-forge and PyPI obsolete now?
But of course this only applies when you use sxml. Artanis apparently can also use plain strings or templates as responses, in which case you will have to take care of safely encoding things too.