MinIO is a fantastic tech and they seemed to be really patient to resolve this issue (waiting 3 years before doing actions). I'll continue to use them and recommend them everywhere I work. They really deserve respect... And to be paid for their hard work.
I worked on Objects at Nutanix for the last ~12 months. Nutanix had originally used the API server in MinIO to translate between the S3 REST API and internal RPCs. MinIO's claim on this blog post that "Nutanix Objects is built around MinIO object storage" is a gross exaggeration.
By the time I joined in June 2021, MinIO was deprecated and we were using an in-house S3 REST API server. I am skeptical that any of the APGL code was distributed because we just weren't using it around the time that MinIO changed from Apache to AGPL.
MinIO is leveraging their switch to the AGPL license as a vehicle to extract immense seven figure plus licensing agreements from "big" companies who relied on non-AGPL versions of the code in the past.
I don't know if that makes MinIO bad, better or good but it's all about money.
Around 2019, a lot of kubernetes distributions started popping up. They often bundle various open source solutions into one platform/PaaS and sell it to the end users. I wonder,
- What are the consequences for these companies?
- Do they share revenue with the open source projects?
- Can they simply distribute these services without any consequences?
- If not, When and How does a small open source project org enforce track and their license?
The consequences is that these companies get very rich and they eventually take-over the open-source project.
See Redis for example, two Israeli dudes took the open-source Redis, made tons of money.
Everyone is happy: the two founders became rich, the VCs became rich.
What about the authors and contributors of Redis ?
Well thank you for the gift. As a present you can have the privilege to work for us to keep maintaining your bugs.
Don't complain too much.
Then you can rewrite the history to make it sound like you created Redis and it's a win, while it's actually just a very smart dude in Italy who wrote most of the software using his own sweat and support from his employer (Pivotal).
> What about the authors and contributors of Redis ? Well thank you for the gift.
He was eventually hired by Redis-the-company, allowed them to use the trademark (originally they were Redis Labs which was a compromise with him), went to their conferences, trained their Redis developers (who contributed to Redis-the-open-source), etc.. I assume he was happy with the deal as he spoke positively about them and chose to spend a lot of time with them, and eventually retired after I presume getting a nice amount of money from the decade-long adventure.
These days liberal OSS licenses are really just free labor for this kind of thing. If you use a very liberal OSS license just make sure you are 100% OK with your work being appropriated this way, including having your name stripped from it and some hustler taking credit.
In the long term I think this kind of behavior is going to kill open source for things beyond libraries and building blocks.
Everything open on the Internet is destroyed by exploitation of one form or another: appropriation, spam, scams, etc. I've become fond of saying "the Internet is a dark forest."
If he didn’t want someone commercializing his software, he should used a different license. His own employer is is a commercial wrapper on an open source Project.
Note that projects in the cloud native space are mostly Apache-licensed. (For example, the CNCF only approves other licenses on an exception basis I believe.) In that case, so long as attributions/trademarks are honored (which oddly seems to not have been the case here), projects licensed in that way can be freely used with Kubernetes without other restrictions.
Companies are not doing anything wrong when they are profiting off a permissive license. Its only a problem if they do something which is not in line with the license.
If people want to make money off open source, then dual licensing is a good way.
Permissively licensing code and then when companies use it as they wish (due to the permissive license), complaining about it does not look good.
Kudos for MinIO team for spending THREE YEARS trying to resolve this. Shame that they had to resort to a public naming and shaming but sometimes corporate entities are tone deaf.
Now this is exposed the next question is if Nutanix Objects is just a MinIO wrapper then what value are they even proving here?
Please don't blame engineers on every single issue. The engineer may not even know there's an issue here. They may be assured by their boss or legal department that they are in the clear. They may not even think about such mundane things like licensing and stuff, that's what they have higher ups for.
If someone is to blame, then it's the company leadership and legal department. As much as we want to make us engineers more important than we are, we are not decision makers. Blame should be put where it belongs.
> They may not even think about such mundane things like licensing and stuff
Imagine a medical doctor or civil engineer claiming that knowing the laws of their professions is "mudane". That's why no one takes programers seriously.
> we are not decision makers.
You totally can decide to not work on stuff you are not comfortable with. It's not like there's a shortage of software engineering jobs.
Software engineers are in a position of great privilege: if we can’t hold ourselves to account, what are we doing? Almost any software engineer put in a difficult position can get up and walk into another job — “it’s not my decision” is not an acceptable excuse for (almost any) software engineer.
Blame lies with those who are complicit by choice, just as much as those who are directing the behaviour.
> They may not even think about such mundane things like licensing and stuff, that's what they have higher ups for.
Oh, come on. Engineers these days are not stupid. While I agree that their boss could plainly lie to them that he bought a commercial license, it was more like, "What will we use for the underlying storage?" "Maybe MiniIO, they're S3-compatible and efficient." "Fine. Can we use their code, though?" "Sure, it's open source, and we are a *aaS business, so no problem." I saw this kind of thinking before.
The engineers might understand the issue better than legal. Most engineers I've meet understand the spirit and intentions of the open source license far better than the legal teams, who are more interested in whether or not you could be successfully sued.
One of the issues with open source software, from a branding perspective, is that you can technically be in the clear, but violate the social contract that implicitly exist in the community. Many companies fail to factor that part in when running licenses through legal.
You are being downvoted but I actually think there are some fair points that you are making.
We use a lot of FOSS in our company. We pay licenses and contribute very little (our job isn't to improve gitlab or docker, we are shipping a software product on top of that), but I wouldn't know where exactly we are in the legal-illegal spectrum to save my life.
I consider myself an employee, not an entrepreneur. If I was an entrepreneur, I sure would happily seek legal advice on what exactly is fair use of open source. But really, I wouldn't know who to trust on the free advice market to figure out what I'm allowed and not allowed to do when starting up. I have absolutely 0 interest in legal stuff and it's mostly scary and confusing to me (and that's probably why I don't do any entrepreneurship, not even a side hustle in consulting), and I wish I and other salary men would be given a break about what the company is doing.
Nutanix shouldn't do what they are doing, but I don't think engineers should be to blame. At the end of the day, if an employee would have to go through everything that the company might not do perfectly right before deciding on a job, we would work nowhere. I wouldn't work for Oracle, but where to draw the line exactly ?
libvirt uses LGPL and the KVM/linux kernel uses GPL. Both are fine to keep to yourself if you run it on your own machine and only expose it over the network.
MinIO uses AGPL which explicitly includes network usage so Nutanix is forced to provide all patches and associated code.
This really seems like Nutanix just didn’t include the MinIO NOTICES file in their OSS disclosures for some reason. Something so minor should have been an easy oversight to fix. Without actually testing out Nutanix, it’s hard to know if they are actually violating this part of the Apache license. MinIO isn’t included in their “open source packages we use” webpage, but that’s not where the NOTICES message would need to be included. Either way, it’s odd that things escalated like this.
The newer AGPL versions of MinIO would offer its own licensing challenge for Nutanix (which is part of the reason for the switch to AGPL). But that’s not even what MinIO is focusing on in their post. MinIO also don’t show the version of their software that they claim Nutanix is using. And it’s very possible that Nutanix froze the minio version in April 2021 (quite likely the case).
Nutanix Objects does not use minio in the core data path. The presence of a binary in a kubernetes pod doesn't necessarily mean that the binary is being used or the fact that nutanix objects is nothing but a wrapper over minio. Earlier implementations did use minio purely as a S3 protocol adapter, i.e a protocol translator from S3 API to Nutanix internal storage protocol. This was something that was publicly acknowledged : https://blocksandfiles.com/2019/11/07/nutanix-objects-storag...
However, in later releases they seemed to have replaced the minio based protocol adapter to something that they developed in-house in C++ and have no longer using minio in their protocol stack.
I recently left Nutanix, but I worked on Objects for the past 12 months. The MinIO path was deprecated by the time that I joined. I don't have enough information to confidently side with either Nutanix or MinIO, but I'll clarify and confirm a few things:
- As you said, MinIO was used to translate S3 REST API requests to internal RPCs.
- MinIO was replaced with an in-house S3 API server.
- I distinctly remember seeing a patch 6-12 months ago where MinIO was removed from the build.
ROFL if you see block and files as the official disclosure fron Nutanix is a great testament of how that company is run :) Try getting their OSD file and see if MinIO is listed :)
Will revoking their license stop Nutanix from using MinIO or will they have to go to court to get them to stop? I don't see any mention of a lawsuit in the post.
My intuition is that they're escalating progressively. Threats and lawsuits, as a general rule, make it more difficult to reach an amicable resolution. I'm inclined to interpret MinIO's response as a mature and prudent one.
>First, revoke the license. That means they are no longer permitted to redistribute the code.
I wonder how it works. What is the act of revoking an open source license exactly? I assume they simply sent a letter and wrote a blog post? Pretty sure in my country it would have no legal force. Is it different in the US?
As soon as they are in violation of the license, they no longer have permission to redistribute the code, because the license is the only thing that allows that and it only allows that if they are in compliance with the license. So there is cause for a lawsuit immediately, not after any other action.
You'd think Nutanix would have the brains to change the names of the deployed binaries, which brings up an interesting question. How do you detect license violation if the violator has replaced the brand name across the codebase?
Alternate interpretation: Nutanix is fully compliant with the terms of Apache 2 and refused to be extorted into paying MinIO money.
The press release is high on FUD (can’t revoke an irrevocable license, no evidence presented they have deployed the AGPLv3 version) and low on details why it took them three years to issue a press release when an injunction would have been granted pretty quick if Nutanix were truly in violation of the Apache license.
I don’t claim to know the details but I do know a little bit the rights under Apache2 and (unless my understanding is incorrect) MinIO’s claims are baffling.
Patterns of strings, function names, other symbols and the entire call graph usually show up in the compiled binary, unless they apply some sort of obfuscator to the process.
Readit News
Until then here is my spicy story: - In 2019: Minio Sales contacted Nutanix (like this user mentioned https://news.ycombinator.com/item?id=32152645) hoping for a nice big cheque.
- 2019-2021: Nutanix cites Apache-2 license and refused to pay.
- 2021: Minio changed its license to AGPL (probably few others like Nutanix)
- 2021: Nutanix knows this and refuses to use AGPL version with their product.
- 2022: Discussion went on for another year and nothing came out from Nutanix.
- Now: Minio decided to publicly shame the company.
By the time I joined in June 2021, MinIO was deprecated and we were using an in-house S3 REST API server. I am skeptical that any of the APGL code was distributed because we just weren't using it around the time that MinIO changed from Apache to AGPL.
I don't know if that makes MinIO bad, better or good but it's all about money.
Deleted Comment
- What are the consequences for these companies?
- Do they share revenue with the open source projects?
- Can they simply distribute these services without any consequences?
- If not, When and How does a small open source project org enforce track and their license?
See Redis for example, two Israeli dudes took the open-source Redis, made tons of money.
Everyone is happy: the two founders became rich, the VCs became rich.
What about the authors and contributors of Redis ? Well thank you for the gift. As a present you can have the privilege to work for us to keep maintaining your bugs. Don't complain too much.
Then you can rewrite the history to make it sound like you created Redis and it's a win, while it's actually just a very smart dude in Italy who wrote most of the software using his own sweat and support from his employer (Pivotal).
He was eventually hired by Redis-the-company, allowed them to use the trademark (originally they were Redis Labs which was a compromise with him), went to their conferences, trained their Redis developers (who contributed to Redis-the-open-source), etc.. I assume he was happy with the deal as he spoke positively about them and chose to spend a lot of time with them, and eventually retired after I presume getting a nice amount of money from the decade-long adventure.
In the long term I think this kind of behavior is going to kill open source for things beyond libraries and building blocks.
Everything open on the Internet is destroyed by exploitation of one form or another: appropriation, spam, scams, etc. I've become fond of saying "the Internet is a dark forest."
If people want to make money off open source, then dual licensing is a good way.
Permissively licensing code and then when companies use it as they wish (due to the permissive license), complaining about it does not look good.
Now this is exposed the next question is if Nutanix Objects is just a MinIO wrapper then what value are they even proving here?
If someone is to blame, then it's the company leadership and legal department. As much as we want to make us engineers more important than we are, we are not decision makers. Blame should be put where it belongs.
Imagine a medical doctor or civil engineer claiming that knowing the laws of their professions is "mudane". That's why no one takes programers seriously.
> we are not decision makers.
You totally can decide to not work on stuff you are not comfortable with. It's not like there's a shortage of software engineering jobs.
Blame lies with those who are complicit by choice, just as much as those who are directing the behaviour.
Oh, come on. Engineers these days are not stupid. While I agree that their boss could plainly lie to them that he bought a commercial license, it was more like, "What will we use for the underlying storage?" "Maybe MiniIO, they're S3-compatible and efficient." "Fine. Can we use their code, though?" "Sure, it's open source, and we are a *aaS business, so no problem." I saw this kind of thinking before.
One of the issues with open source software, from a branding perspective, is that you can technically be in the clear, but violate the social contract that implicitly exist in the community. Many companies fail to factor that part in when running licenses through legal.
We use a lot of FOSS in our company. We pay licenses and contribute very little (our job isn't to improve gitlab or docker, we are shipping a software product on top of that), but I wouldn't know where exactly we are in the legal-illegal spectrum to save my life.
I consider myself an employee, not an entrepreneur. If I was an entrepreneur, I sure would happily seek legal advice on what exactly is fair use of open source. But really, I wouldn't know who to trust on the free advice market to figure out what I'm allowed and not allowed to do when starting up. I have absolutely 0 interest in legal stuff and it's mostly scary and confusing to me (and that's probably why I don't do any entrepreneurship, not even a side hustle in consulting), and I wish I and other salary men would be given a break about what the company is doing.
Nutanix shouldn't do what they are doing, but I don't think engineers should be to blame. At the end of the day, if an employee would have to go through everything that the company might not do perfectly right before deciding on a job, we would work nowhere. I wouldn't work for Oracle, but where to draw the line exactly ?
I wonder how many other licences they're violating this way.
MinIO uses AGPL which explicitly includes network usage so Nutanix is forced to provide all patches and associated code.
https://github.com/minio/minio/commits/master/LICENSE
This really seems like Nutanix just didn’t include the MinIO NOTICES file in their OSS disclosures for some reason. Something so minor should have been an easy oversight to fix. Without actually testing out Nutanix, it’s hard to know if they are actually violating this part of the Apache license. MinIO isn’t included in their “open source packages we use” webpage, but that’s not where the NOTICES message would need to be included. Either way, it’s odd that things escalated like this.
The newer AGPL versions of MinIO would offer its own licensing challenge for Nutanix (which is part of the reason for the switch to AGPL). But that’s not even what MinIO is focusing on in their post. MinIO also don’t show the version of their software that they claim Nutanix is using. And it’s very possible that Nutanix froze the minio version in April 2021 (quite likely the case).
However, in later releases they seemed to have replaced the minio based protocol adapter to something that they developed in-house in C++ and have no longer using minio in their protocol stack.
- As you said, MinIO was used to translate S3 REST API requests to internal RPCs.
- MinIO was replaced with an in-house S3 API server.
- I distinctly remember seeing a patch 6-12 months ago where MinIO was removed from the build.
Read Apache v2 attribution clauses.
But the parent to your comment says they don't use it (any more)???
If they then continue to redistribute it, they are committing a copyright violation. That’s when there is cause for a lawsuit.
I wonder how it works. What is the act of revoking an open source license exactly? I assume they simply sent a letter and wrote a blog post? Pretty sure in my country it would have no legal force. Is it different in the US?
There's all kinds of specific legal teeth for that behavior.
The press release is high on FUD (can’t revoke an irrevocable license, no evidence presented they have deployed the AGPLv3 version) and low on details why it took them three years to issue a press release when an injunction would have been granted pretty quick if Nutanix were truly in violation of the Apache license.
I don’t claim to know the details but I do know a little bit the rights under Apache2 and (unless my understanding is incorrect) MinIO’s claims are baffling.
Deleted Comment