Readit News logoReadit News
victormy · 2 months ago
Big thanks to MinIO, RustFS, and Garage for their contributions. That said, MinIO closing the door on open source so abruptly definitely spooked the community. But honestly, fair play to them—open source projects eventually need a path to monetization.

I’ve evaluated both RustFS and Garage, and here’s the breakdown:

Release Cadence: Garage feels a bit slower, while RustFS is shipping updates almost weekly.

Licensing: Garage is on AGPLv3, but RustFS uses the Apache license (which is huge for enterprise adoption).

Stability: Garage currently has the edge in distributed environments.

With MinIO effectively bowing out of the OSS race, my money is on RustFS to take the lead.

ahepp · 2 months ago
> open source projects eventually need a path to monetization

I guess I'm curious if I'm understanding what you mean here, because it seems like there's a huge number of counterexamples. GNU coreutils. The linux kernel. FreeBSD. NFS and iSCSI drivers for either of those kernels. Cgroups in the Linux kernel.

If anything, it seems strange to expect to be able to monetize free-as-in-freedom software. GNU freedom number 0 is "The freedom to run the program as you wish, for any purpose". I don't see anything in there about "except for business purposes", or anything in there about "except for businesses I think can afford to pay me". It seems like a lot of these "open core" cloud companies just have a fundamental misunderstanding about what free software is.

Which isn't to say I have anything against people choosing to monetize their software. I couldn't afford to give all my work away for free, which is why I don't do that. However, I don't feel a lot of sympathy to people who surely use tons of actual libre software without paying for it, when someone uses their libre software without paying.

mikestorrent · 2 months ago
I think, if anything, in this age of AI coding we should see a resurgence in true open-source projects where people are writing code how they feel like writing it and tossing it out into the world. The quality will be a mixed bag! and that's okay. No warranty expressed or implied. As the quality rises and the cost of AI coding drops - and it will, this phase of $500/mo for Cursor is not going to last - I think we'll see plenty more open source projects that embody the spirit you're talking about.

The trick here is that people may not want to be coding MinIO. It's like... just not that fun of a thing to work on, compared to something more visible, more elevator-pitchy, more sexy. You spend all your spare time donating your labour to a project that... serves files? I the lowly devops bow before you and thank you for your beautiful contribution, but I the person meeting you at a party wonder why you do this in particular with your spare time instead of, well, so many other things.

I've never understood it, but then, that's why I'm not a famous open-source dev, right?

elzbardico · 2 months ago
I don't think there's still someone actively working on the Linux kernel without receiving a salary, and this for the last two decades more or less.
lima · 2 months ago
Shipping updates almost weekly is the opposite of what I want for a complex, mission-critical distributed system. Building a production-ready S3 replacement requires careful, deliberate and rigorous engineering work (which is what Garage is doing[1]).

It's not clear if RustFS is even implementing a proper distributed consensus mechanism. Erasure Coding with quorum replication alone is not enough for partition tolerance. I can't find anything in their docs.

[1]: https://arxiv.org/pdf/2302.13798

peterashford · 2 months ago
"open source projects eventually need a path to monetization"

Why?

elzbardico · 2 months ago
Human beings have this strange desire to be fed, have shelter and other such mundane stuff, all of those clearly less important than software in the big scheme of things, of course.
drnick1 · 2 months ago
The beauty of open source is that there are all kinds of reasons for contributing to it, and all are valid. For some, it's just a hobby. For others, like Valve, it's a means of building their own platform. Hardware manufacturers like AMD (and increasingly Nvidia) contribute drivers to the kernel because they want to sell hardware.
victormy · 2 months ago
I believe that, at the end of the day, open source enthusiasts still need to make a living.
snthpy · 2 months ago
Thanks. I hadn't heard of RustFS. I've been meaning to migrate off my MinIO deployment.

I recently learned that Ceph also has an object store and have been playing around with microceph. Ceph also is more flexible than garage in terms of aggregating differently sized disks. Since it's also already integrated in Proxmox and has over a decade of enterprise deployments, that's my top contender at the moment. I'm just not sure about the level of S3 API compatibility.

Any opinions on Ceph vs RustFS?

lima · 2 months ago
Ceph is quite expensive in terms of resource usage, but it is robust and battle-tested. RustFS is very new, very much a work in progress[1], and will probably eat your data.

If you're looking for something that won't eat your data in edge cases, Ceph (and perhaps Garage) are your only options.

[1]https://github.com/rustfs/rustfs/issues/829

boomskats · 2 months ago
RustFS vs MinIO latest performance comparisons here: https://github.com/rustfs/rustfs/issues/73#issuecomment-3385...
raxxorraxor · 2 months ago
> open source projects eventually need a path to monetization.

I don't think open source projects need a path to monetization in all cases, most don't have that. But if you make such a project your main income, you certainly need money.

If you then restrict the license, you are just developing commercial software, it then has little to do with open source. Developing commercial software is completely fine, but it simply isn't open source.

There is also real open source software with a steady income and they are different than projects that change to commercial software and we should discriminate terms here.

antman · 2 months ago
SeaweedFS with S3 API? Differentiates itself with claims of ease of use and small files optimization
RealStickman_ · 2 months ago
Last time I checked (~half a year ago) Garage didn't have a bunch of s3 features like object versioning and locking. Does RustFS have a list of s3 features they support?
lmc · 2 months ago
Any idea who is behind RustFS?
cpach · 2 months ago
Good question. On their website they list 3550 Lenox Road, NE Atlanta, Georgia 30326 as their address. But no info about the company name, CEO or anything like that.
gethly · 2 months ago
There is https://github.com/seaweedfs/seaweedfs

I haver not used it but will be likely a good minio alternative for people who want to run a server and don't use minio just as s3 client.

chrislusf · 2 months ago
This is Chris and I am the creator of SeaweedFS. I am starting to work full time on SeaweedFS now. Just create issues on SeaweedFS if any.

Recently SeaweedFS is moving fast and added a lot more features, such as: * Server Side Encryption: SSE-S3, SSE-KMS, SSE-C * Object Versioning * Object Lock & Retention * IAM integration * a lot of integration tests

Also, SeaweedFS performance is the best in almost all categories in a user's test https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... And after that, there is a recent architectural change that increases performance even more, with write latency reduced by 30%.

written-beyond · 2 months ago
Congratulations on earning that opportunity!

Thank you for your work. I was in a position where I had to choose between minio and seaweed FS and though seaweed FS was better in every way the lack of an includes dashboard or UI accessibility was a huge factor for me back then. I don't expect or even want you to make any roadmap changes but just wanted to let you know of a possible pain point.

lima · 2 months ago
Is it stable now? Last time I checked, the amount of correctness bugs being fixed in the Git history wasn't very confidence-inspiring.
rednb · 2 months ago
Since storage is a critical component, I closely watched it and engaged with the project for about 2 years circa as i contemplated adding it to our project, but the project is still immature from a reliability perspective in my opinion.

No test suite, plenty of regressions, and data loss bugs on core code paths that should have been battled tested after so many years. There are many moving parts, which is both its strength and its weakness as anything can break - and does break. Even Erasure Coding/Decoding has had problems, but a guy from Proton has contributed a lot of fixes in this area lately.

One of the big positive in my opinion, is the maintainer. He is an extremely friendly and responsive gentleman. Seaweedfs is also the most lightweight storage system you can find, and it is extremely easy to set up, and can run on servers with very little hardware resources.

Many people are happy with it, but you'd better be ready to understand their file format to fix corruption issues by hand. As far as i am concerned, i realized that after watching all these bugs, the idea of using seaweedfs was causing me more anxiety than peace of mind. Since we didn't need to store billions of files yet, not even millions, we went with creating a file storage API in ASP.NET Core in 1 or 2 hours, hosted on a VPS, that we can replicate using rsync without problem. Since i made this decision, i have peace of mind and no longer think about my storage system. Simplicity is often better, and OSes have long been optimized to cache and serve files natively.

If you are not interested in contributing fixes and digging into the file format when a problem occurs, and if your data is important to you, unless you operate at the billions of files scalability tier Seaweedfs shines at, i'd suggest rolling your own boring storage system.

yahooguntu · 2 months ago
We're in the process of moving to it, and it does seem to have a lot of small bugfixes flying around, but the maintainer is EXTREMELY responsive. I think we'll just end up doing a bit of testing before upgrading to newer versions.

For our use case (3 nodes, 61TB of NVMe) it seems like the best option out of what I looked at (GarageFS, JuiceFS, Ceph). If we had 5+ nodes I'd probably have gone with Ceph though.

nodesocket · 2 months ago
I'm looking at deploying SeaWeedFS but the problem is cloud block storage costs. I need 3-4TB and Vultr costs $62.50/mo for 2.5TB. DigitalOcean $300/mo for 3TB. AWS using legacy magnetic EBS storage $150/mo... GCP persistent disk standard $120/mo.

Any alternatives besides racking own servers?

*EDIT* Did a little ChatGPT and it recommended tiny t4g.micro then use EBS of type cold HDD (sc1). Not gonna be fast, but for offsite backup will probably do the trick.

xyzzy123 · 2 months ago
I'm confused why you would want to turn an expensive thing (cloud block storage) into a cheaper thing (cloud object storage) with worse durability in a way that is more effort to run?

I'm not saying it's wrong since I don't know what it's for, I'm just wondering what the use-case could be.

Bombthecat · 2 months ago
Hetzner VM with mounted storage box.

https://www.hetzner.com/storage/storage-box/

It's not as fast as local storage of course, but it's cheap!

huntaub · 2 months ago
Shot you an email about how we can potentially help you with this.
hobofan · 2 months ago
SeaweedFS has been our choice as a replacement for both local development and usage in our semi-ephemeral testing k8s cluster (both for its S3 interface). The switch went very smooth.

I can't really say anything about advanced features or operational stability though.

lionkor · 2 months ago
Sadly there's nothing in the license of seaweedFS that would stop the maintainer from pulling a minio -- and this time without breaking the (at least spirit of the) terms of the project's license.

Not an issue at all until they do.

spicymaki · 2 months ago
Stallman was right. When will the developer community learn not to contribute to these projects with awful CLAs. The rug has been pulled.
pabs3 · 2 months ago
MinIO doesn't seem to have had a CLA though?
EgoIncarnate · 2 months ago
MinIO had a de facto CLA. MinIO required contributors to license their code to the project maintainers (only) under Apache 2. Not as bad as copyright assignment, but still asymmetric (they can relicense for commercial use, but you only get AGPL). https://github.com/minio/minio/blob/master/.github/PULL_REQU...
dzogchen · 2 months ago
Where does Stallman say anything about CLAs?
creatonez · 2 months ago
Except... the FSF is actually on the extreme opposite end of this issue. They do formal copyright assignment from the GNU contributors to the FSF. This way, they have a centralized final say on enforcement that is resistant to copyleft trolls, but it ultimately allows the theoretical possibility of a rugpull.
reedciccio · 2 months ago
The FSF can't pull the rug because of its bylaws
cantagi · 2 months ago
They have been removing features from the open source version for a while.

The closest alternative seems to be RustFS. Has anyone tried it? I was waiting until they support site replication before switching.

bityard · 2 months ago
Garage is a popular alternative to Minio. https://garagehq.deuxfleurs.fr

I hadn't heard of RustFS and it looks interesting, although I nearly clicked away based on the sheer volume of marketing wank on their main page. The GitHub repo is here: https://github.com/rustfs/rustfs

adamcharnock · 2 months ago
We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

dalenw · 2 months ago
I use garage at home, single node setup. It's very easy and fast, I'm happy with it. You're missing out on a UI for it, but MountainDuck / CyberDuck solves that problem for me.
eproxus · 2 months ago
Yeah, that page is horrendous and looks super sketchy. It looks like a very professional fishing attempt to get unsuspecting developers to download malware.

They have a lot of obviously fake quotes from non-existent people at positions that don’t even mention what company it is. The pictures are misgendered and even contain pictures of kids.

Feels like the whole page is AI generated.

victormy · 2 months ago
Speaking as an open-source enthusiast, I’m actually really digging RustFS. Honestly, anything that can replace or compete with MinIO is a win for the users. Their marketing vibe feels pretty American, actually—they aren't afraid to be loud and proud, haha. You gotta give it to them though, they’ve got guts, and their timing is spot on.
cromka · 2 months ago
I saw an article here not long about where someone explained they were hosting their Kopia or Nextcloud aver Garage, but I can't find it anymore.

This was going to be my next project, as I am currently storing my Kopia/Ente on MinIO in a non-distributed way. MinIO project going to shi*s is a good reason to take on this project faster than later.

nikeee · 2 months ago
I maintain an S3 client that has a test matrix for the commonly used S3 implementations. RustFS regularly breaks it. Last time it did I removed it from the matrix because deleteObject suddenly didn't delete the object any more. It is extremely unstable in its current form. The website states that it is not in a production-ready state, which I can confirm.

I'd take a look at garage (didn't try seaweed yet).

edude03 · 2 months ago
> I maintain an S3 client that has a test matrix for the commonly used S3 implementations.

Is it open to the public? I'd like to check it out

positisop · 2 months ago
If it is not an Apache/CNCF/LinuxFoundation project, it can be a rug pull aimed at using open source for getting people in the door only. They were never open for commits, and now they have abandoned open source altogether.
victormy · 2 months ago
The Good: Single-node is stable, and the team moves fast—most of my reported bugs get patched within a couple of weeks. The Bad: Distributed mode needs work. Bucket replication and lifecycle policies are still WIP (as noted in their roadmap) and not usable yet.

It's promising, but definitely check the roadmap before deploying at scale.

rbartelme · 2 months ago
Might be coming soon based on this: https://docs.rustfs.com/features/replication/
maxloh · 2 months ago
Although promising, RustFS is a Chinese product. This would be a non-starter for many.
jasonvorhe · 2 months ago
Because they aren't thinking about all the chinese wetware they'd be writing down that decision with.
PunchyHamster · 2 months ago
From what I looked still very fresh project, to the point running out of date minio version will most likely be less problematic than latest rustfs
pankajdoharey · 2 months ago
Sad to see these same people were behind GlusterFS.
mbreese · 2 months ago
Well, maybe they are using that experience to build something better this time around? One can hope...
js4ever · 2 months ago
I made recently an open source alternative to minio Server & minio UI also in Rust:

https://github.com/vibecoder-host/ironbucket/

https://github.com/vibecoder-host/ironbucket-ui

syabro · 2 months ago
Probably just me but I would stay away from anything saying vibecoder in the repo
uroni · 2 months ago
I've been working on https://github.com/uroni/hs5 as a replacement with similar goals to early minio.

The core is stable at this point, but the user/policy management and the web interface is still in the works.

giancarlostoro · 2 months ago
Looks like you cleanly point out their violation of the AGPL. I wish I were a lawyer with nothing better to do, I'd definitely be suing the MinIO group, there's no way they can cleanly remove the AGPL code outsiders contributed.
mbreese · 2 months ago
I don't think there would be an issue with removing AGPL contributed code. You can't force someone to distribute something they don't want to. IANAL, but I believe that what (all?) copyright in software is most concerned with is the active distribution of code -- not the removal of code.

That said, if there was contributed AGPL code, they couldn't change the license on that part of the code w/o a CLA. AGPL also doesn't necessarily mean you have to make the code publicly available, just available to those that you give the program to (I'm assuming AGPL is like the GPL in this regard).

So, that I'd be curious about it is -- (1) is there any contributed AGPL code in the current version? (2) what license is granted to customers of the enterprise version?

Minio can completely use whatever license they want for their code. But, if there was contributed code w/o a CLA, then I'm not sure how a commercial/enterprise license would play with contriubuted AGPL code. It would be an interesting question to find out.

bityard · 2 months ago
I don't see a contributor licensing agreement (CLA), so you may be right.

(I personally choose not to contribute to projects with CLAs, I don't want my contributions to become closed-source in the future.)

EgoIncarnate · 2 months ago
MinIO had a de facto CLA. MinIO required contributors to license their code to the project maintainers (only) under Apache 2. Not as bad as copyright assignment, but still asymmetric (they can relicense for commercial use, but you only get AGPL). https://github.com/minio/minio/blob/master/.github/PULL_REQU...
uroni · 2 months ago
I'm not a contributor to Minio. This is its own separate thing.

I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point and it did look pretty bleak -- it is a civil suit where you have to show damages etc. but IANAL.

I did not like the FUD they were spreading about AGPL at the time since it is a good license for end-user applications.

sph · 2 months ago
Good time to post a Show HN for your project then
bityard · 2 months ago
Interesting! I like the relative simplicity and durability guarantees. I can see using this for dev and proof of concept. Or in situations where HA/RAID are handled lower in the stack.

What is the performance like for reads, writes, and deletes?

And just to play devil's advocate: What would you say to someone who argues that you've essentially reimplemented a filesystem?

uroni · 2 months ago
It uses LMDB, so if the object mapping fits in memory that should be pretty optimal for reading, while using the build-in Linux page cache and not a separate one (important for testing use cases). For write/deletes it has a bit of write-amplification due to the copy-on-write btree. I've implemented a separate, optional WAL for this and also a mode where writes/delete can be bundeled in a transaction, but in practice I think the performance difference should not matter.

W.r.t. filesystem: Yes, aware of this. Initially used minio and also implemented the use case directly on XFS as well and only had problems at larger scales (that still fit on a machine) with it. Ceph went into a similar direction with BlueStore (reimplementing the filesystem, but with RocksDB).

MrZander · 2 months ago
I wish I knew about this last week. I spent way too long trying out MinIO alternatives before getting SeaweedFS to work, but it is overkill for my purposes.

Looks like a great alternative.

liviux · 2 months ago
Fork in Linux foundation incoming. Minio will revert in 1-2 years, but too late, community will move on and never return, reputation lost forever
phoronixrly · 2 months ago
orphea · 2 months ago

  > you may be violating AGPLv3 if you are using MinIO to build commercial products or using it for commercial purposes
Yeah, this is bullshit. I wish the guy used his own advise and spoke to a lawyer :)

speedgoose · 2 months ago
Oh no, I used MinIO once or twice for some unlicensed software.

Should I contact a MinIO salesman to purchase an enterprise license ASAP or is it fine if I license my kids and advent of code solutions under the AGPLv3 license ?

ahepp · 2 months ago
Wait, what's the consensus on this? Are they saying that using object storage over a standard network API which they didn't even create, makes your application a derivative work of the object store?

Or just that the users would need to make minio sources, including modifications, freely available?

I guess that's kind of the big question inherent to the AGPL?

spapas82 · 2 months ago
Minio is more or less feature complete for most use cases. Actually the last big update of minio removed features (the UI). I am using minio for 5 years and haven't messed with it or used any new thingie for the last 5 years (i.e since I installed it); I only update to new versions.

So if the minio maintainers (or anybody that forks the project and wants to work it) can fix any security issues that may occur I don't see any problems with using it.

cromka · 2 months ago
> Actually the last big update of minio removed features (the UI)

AFIK they removed it only to move it to their paid version, didn't they?

spapas82 · 2 months ago
Well I didn't mind when they removed it and certainly I didn't consider their paid version which is way too expensive for most use cases.

The UI was useful when first configuring the buckets and permissions; if you've got it working (and don't need to change anything) you're good to go. Also, everything can be configured without the UI (not so easily of course).

lionkor · 2 months ago
yes
deeebug · 2 months ago
> So if the minio maintainers (or anybody that forks the project and wants to work it) can fix any security issues that may occur I don't see any problems with using it.

The concerning language for me is this part that was added:

> Critical security fixes may be evaluated on a case-by-case basis

It seems to imply that any fixes _may_ be merged in, but there's no guarantees.

spapas82 · 2 months ago
Yes this is concerning for me too. Hopefully if they don't fix/merge security issues somebody will fork and maintain it. It shouldn't be too much work. I'd even do it myself if I was experienced in golang.
fithisux · 2 months ago
I used it for my experiments in Docker. I once or two used the UI, I always connected through python.
aftbit · 2 months ago
Shocker... they abandoned POSIX compatibility, built a massively over-complicated product, then failed to compete with things like Ceph on the metal side or ubiquitous S3/R2/B2 on the cloud side.
PunchyHamster · 2 months ago
No, they rebranded to AIStor and are now selling to AI companies.

Minio is/was pretty solid product for places where rack of servers for Ceph wasn't an option (it does have quite a bit higher memory requirements), or you just need a bit of S3 (like we have small local instances that just run as build cache for CI/CD)

But that's not where money is

throwaway894345 · 2 months ago
> they abandoned POSIX compatibility, built a massively over-complicated product

This is a wild sentence--how can you criticize them for abandoning POSIX support __and__ building a massively over-complicated product? Making a reliable POSIX system is inherently very complex.

bee_rider · 2 months ago
I think the criticism (just interpreting the post, don’t know anything about the technical situation) is that the complication is not necessary/worthwhile.

POSIX can be complicated, but it puts you in a nice ecosystem, so for some use-cases complex POSIX support is not over complicated. It is just… appropriately complicated.

Deleted Comment

Deleted Comment

ahepp · 2 months ago
What would go in to POSIX compatibility for a product like this which would make it complicated? Because the kind of stuff that stands out to me is the use of Linux specific syscalls like epoll/io_uring vs trad POSIX stuff like poll. That doesn't seem too complicated?
dark-star · 2 months ago
S3 object names are not POSIX compatible.

"foo" and "foo/bar" are valid S3 object names that cannot coexist on a POSIX filesystem

ahepp · 2 months ago
So when we say "they abandoned posix compatibility", are we saying "They abandoned the POSIX filesystem storage backend"? I believe that's true, I used to use minio on a FreeBSD server but after an update I had to switch to just passing in zfs block devs.

Or are we saying that they no longer support running minio on POSIX systems at all, due to using linux specific syscalls or something else I'm not thinking of? I don't know whether they did this or not.

Those seem like two very different things to me, and when someone says "they don't support POSIX", I assume the latter