> One option is through the usage of UBI container images which are based on RHEL and available from multiple online sources (including Docker Hub). Using the UBI image, it is easily possible to obtain Red Hat sources reliably and unencumbered. We have validated this through OCI (Open Container Initiative) containers and it works exactly as expected.
> Another method that we will leverage is pay-per-use public cloud instances. With this, anyone can spin up RHEL images in the cloud and thus obtain the source code for all packages and errata. This is the easiest for us to scale as we can do all of this through CI pipelines, spinning up cloud images to obtain the sources via DNF, and post to our Git repositories automatically.
That's quite the workaround, the rocky team has proven it's willing to get hacky if needed.
The public cloud route is pretty elegant. Red Hat is restricting source code to subscribers only, so Rocky contributors will just subscribe for an hour at a time when they need to download source code. There’s no way for Red Hat to stop this without terminating all public cloud licensing everywhere.
It sounds like they have two different mechanisms they can pull from currently, which will get them to parity with RHEL releases.
Red Hat would need to shift a few knobs and probably offend quite a few people running UBI images at least (including a zillion folks in the OpenShift community who rely on them) to cut off this current approach to getting the sources.
I wonder if Red Hat is willing to play this game of whack a mole? And IMO, was it worth it?
I suspect they're playing with unintended consequences now.
One of the nice "features" of buying CentOS, is that it meant CentOS was never going to compete - there was a clear line between community and professional, and CentOS were never going to sell you professional services.
Pushing everyone to non-RH builds has removed that line, and there's a strong chance that non-RH builds selling professional services, is going to have a higher opportunity-cost than publishing CentOS did.
That wasn't relevant in buying CentOS, plenty of people were selling personal services for CentOS. IBM itself was doing it, and perhaps Kyndryl is still doing the same for the newfangled RHEL rebuilds.
And even now, technically RESF is the one producing the distro and it's also not selling professional services. Who produces the distro has no effect on who sells the services.
1) having something to show their customers who has the actual expertise
2) making it clear that the Red Hat of today is significantly more open than the Red Hat of 2014 when neither CentOS Stream nor UBI existed and CentOS releases were months late despite the SRPMs being on ftp.redhat.com
3) making it obvious that they are respecting the GPL, and that no one gives a flying f**k about "free as in freedom" because all the uprising was always about either the free beer or the clicks/likes.
Could they not just leave these alternate channels a few weeks or months behind the current release that only the subscribers have access to? That would keep them as the most current, up-to-date source over Rocky Linux.
"Approximately the same lag as you used to get with CentOS" would seem pretty fair to me - I'm aware people grumbled about it and understandably so, but it was still a relatively stable and relatively co-operative situation.
I feel like returning to that apparent Schelling Point could quite easily be an improvement over the Red Queen's Race that I worry is developing here.
Here's a thought, maybe Red Hat was being honest when they said that they were not under an obligation to make it easy for rebuilders, and that's it? Maybe they weren't out to immediately kill the clones because they know that they can't? We have actually heard very little communication from Red Hat most of it has been speculation from people on what they might or could do, but as you point out there are ways around the changes Red Hat made.
Honestly this from this post Rocky conflicts with the "RHEL is closed source/proprietary/paywalled" narrative that people are trying to push. If RHEL was truly any of those things Rocky wouldn't have been able to continue on, but they were able to quickly find a solution, though to me it seems a bit hacky. If Rocky is pulling packages from the supposedly untested, beta of RHEL CentOS Stream, and UBI and some EC2 instance, why would I use that over something that was build cohesively in one place like Stream?
They are actually under exactly that obligation. It's very explicit in the gpl not only what the terms are, but what their intent is, precisely so that no one can ever claim any other possible interpretation.
Mike McGrath has been very explicit about this in his comments on the Ask Noah Show podcast episode[1] and a number of responses in the r/Linux subreddit.
The solution here is forking and accepting that IBM just doesn't want to share. The whole value of the Red Hat eco system is lots of people using the down stream variants. Actual direct licensees of Red Hat are not where most of the action is.
The value creation is actually distributed across the ecosystem. People encounter issues, report them, and fixes are distributed. If you break that cycle and get IBM out of the loop, the process just continues elsewhere. The vast majority of that ecosystem does not pay IBM a single dollar and probably never will.
IBM is a company that is in slow decline, so the remaining Red Hat employees are facing an extended period of that company just squeezing harder and harder until nothing remains. It's death by a thousand cuts. But the bottom line is that a lot of people doing the hard work of committing actual code on behalf of the ecosystem that are currently employed there will be facing endless rounds layoffs, reorganizations, restructurings, etc. If there isn't an employee exodus happening there already, that might soon start to happen. The only question is where those people will end up.
A well funded foundation maintaining the fork of the distribution formerly known as Red Hat could be a nice destination for such people. Between Amazon, Oracle, and the countless users of Rocky, Alma, Centos, Fedora, etc. there should be plenty of brain power, motivation, and money to make that happen. They need a stable foundation. They don't need IBM to be part of that.
Don't play IBM's game on their terms. Just cut them loose. They get to do whatever they want downstream, not upstream. And they get to contribute all their fixes under GPL. That's not optional. This foundation would be free to use these fixes as they need to. Comparability with IBM's downstream distribution should not be a goal for this foundation. And if IBM wants to pretend they can do it all by themselves, they are more than welcome to try.
My prediction is that if the big supporters of this ecosystem join forces and do this, IBM will grumble a bit and then ultimately join the foundation because their alternative will be just writing off the investment they made in Red Hat and watch from the sidelines how most of the ecosystem stops depending on IBM's Red Hat.
> IBM will grumble a bit and then ultimately join the foundation because their alternative will be just writing off the investment they made in Red Hat and watch from the sidelines how most of the ecosystem stops depending on IBM's Red Hat.
You could have said that if they switched to using CentOS Stream, and that would even have been my favorite outcome as a Red Hat employee.
However, Rocky Linux is neither a sibling nor a fork of RHEL. It's a debranded clone that by definition cannot even have a single bugfix that isn't in RHEL. For Oracle it's okay because it's peanut money in order to annoy Red Hat, so they can afford this; for Amazon or Facebook it's no good and that's why they forked upstream at the Fedora or CentOS Stream level.
As long as Rocky Linux stays a RHEL rebuild built by a third party like the CentOS of 2010 (except backed by corporate money rather than a guy in Nebraska), Red Hat is already putting millions into "the foundation". That's what they pay for the thousand people that develop Rocky Linux, ahem RHEL. Without them, there can be no Rocky Linux at all. So, as long as Rocky's money making side keeps undermining Red Hat's money making side, game theory predicts no other outcome than death for both RHEL and Rocky.
_EDIT_: if you downvote, I'd be very glad to learn where I'm wrong
And without the GPL and all the code that came with it, there would be no RH. Rocky is making use of the same legal protection that RH is. Yes, RH spends more on development but they are doing so using the tools and existing codebase given to them, for free, by others.
Rocky is doing something no different than what RH is doing, and if this is problematic for RH's hopes of sucking in a few $B, that's more a problem with RH's business model than it is Rocky's. They have made $Bs selling support for free software, some large part of which they didn't author, and now they want to squeeze the entire ecosystem for more.
I have zero sympathy for RH and fully support Rocky's approach here. This is a problem of RH's own creation and trying to deflect the blame onto Rocky is absurd.
Ideally, Rocky's money making side could come to some sort of agreement to share revenue with RHEL's money making side, so that in turn RHEL's software making side doesn't mind sharing code with Rocky's (lack of) software making side.
I think the abundant corporate sponsoring of Rocky Linux proves that AWS, Google, Facebook etc. are happy to pay for access to RedHat's work. They just do not agree to the way that the licensing scales (both costs and hassle).
> It's a debranded clone that by definition cannot even have a single bugfix that isn't in RHEL.
Not necessarily. They could easily have an optional repository for "bugfixes that aren't in RHEL". Those who want bug-for-bug compatibility with RHEL for some reason could simply not enable that repository.
When I was at IBM and RedHat did the "CentOS"-move, IBM-execs was actually pretty pissed off. It was bad optics and RedHat did it on their own, while IBM got "the blame". This is probably more of the same stuff. They think they can get away with being asshats and people will just blame IBM.
We see you, RedHat. You are NOT on the right path.
I originally had a strong anti-RedHat response to this change. When I thought about it and heard RH's response their sharp change makes sense.
They sell RHEL. It's from what I gather their main source of income. Revenue from this funds things like SystemD, a lot of work in Gnome, many many things that RHEL customers and other users of Linux and desktop Linux benefit from. Of course many contributions to open source/GNU tools come from folks in no way affiliated or paid by RH and RH does use these packages but RH also provides a lot of value.
So it stands to reason, to me at least, that to allow anyone to reskin/respin/or basically just ship a RHEL clone without RH branding that is "100% bug/binary compatible with RHEL" just without the license cost is giving away something you work on for free. No rational business would allow this.
CentOS, Fedora are free. RHEL is not. Makes sense.
Historically they've generally politely ignored community rebuilders and got a trifle enervated by commercial rebuilders - they changed how they handled distributing kernel code (to a fully patched tree rather than a pristine tree and a stack of patches I -think- from memory) in response to Oracle doing a commercial rebuild.
Exactly what the triggering incident was this time they've been very careful not to officially say (which is likely a better option than the optics of getting into a finger pointing war with a smaller target), and I suspect we won't be able to fully judge their motivations unless/until the details leak and/or are inferred by people close enough to the situation to guess correctly.
I don't know for sure but if I had to guess early Linux some 25-years ago didn't have the prevalence of polish it has now at least on the desktop. In the server space it was probably solid. That being said businesses back then were likely leery of running a RH clone with just some Linux staff -- better to pay RH for support.
Nowadays one could probably lean on staff to manage issues and arbitrage that go-it-alone mindset over paying the RH subscription. Now that loop-hole is closed.
They sell software though that is mostly licensed under the GPL which has certain requirements regardless of if you give it a way free , support down stream projects financially or make an entire public company around it. I’m not sure if red hat can really stop rebuilds beyond ensuring all their trademarks are removed any more than Debian could suddenly stop ubuntu from using that as their starting point OS. RHEL is not free but as I understand it you are getting support from redhat, potentially some legal protections for another SCO type lawsuit, but you are not paying for Linux. I can’t license red hat Linux and get a non GPL license from red ha can I?
Are the RHEL SRPMS watermarked for individual Customers in any way? It seems like Redhat has no mechanism to stop a torrent of the SRPMS showing-up. Attribution would be exceedingly difficult. Since distribution of FLOSS-licensed source isn’t copyright infringement it’s not like they could DMCA it away.
Arguably the specfiles are able to be copyrighted. I wonder what the license is like for those.
Not no mention, source is text. Plain diff shows any differences. There is a lot of text, and so a lot of haystack to make small changes, but no way to hide them at all, and no way to break the product if the diffs are undiffed or further diffed to obscure the origin.
RH can't actually do what their trying to do, but that's less important than the fact that they want to. I can't see voluntarily having anything to do with them now. I see no value in any product or service they might offer that is worth knowingly working with someone who has exposed such a lack of integrity.
I was poking around the Rocky Linux website, and wondering where to download the latest source code for Rocky 9.2? Let's say IBM decides not to burn up the ecosystem, will Oracle / Alma start using the source that Rocky exfiltrates?
Related question, as a Red Hat subscriber can I still distribute Red Hat ISO and source code? It seems like I should be able to distribute ISO images and source after obtaining them.. but not repackage it? I don't plan to impose any restrictions on the people I distribute to.
Red Hat can't stop you from exercising your rights under the GPL to redistribute the code. You also can't compel them to do business with you. They've structured their support agreements such that if you do exercise your rights under the GPL they will stop supporting you (and decline to offer you future subscriptions).
The value proposition for RHEL is ostensibly support (and a "throat to choke", for whatever that's actually worth). Red Hat's gamble is that no "legitimate" Red Hat subscriber would risk their support entitlement (and the ability to contract with Red Hat for support in the future) by exercising their rights under the GPL.
It's a clever hack. It runs counter to ideals of Free software (and I find it personally repugnant) but it's clever.
To be clear, it's not yet known for certain if the chilling effect of a terminated contract violates the "no further restrictions" clause of the GPL or not. Evidently IBM's lawyers think it doesn't. But it would be good to test it in court first.
Much like GPLv3 was written to counter tivoization, I’d love a GPLv4 to kill what RedHat is doing before they do irreparable the entire free software ecosystem.
In the IBM/RH blog post it references[1] the word appears once, disparagingly, in the gratis sense.
I appreciate the beer / speech distinction can get tiring to explain repeatedly, but it feels like the move to distance themselves from the deeper implications & obligations of free is, shall we say, very carefully calculated.
Could equally be a move to minimise how much of the discussion that springs up around this blog post gets derailed and eaten alive by arguments and/or misunderstandings around said distinction.
Though I suspect we'll both end up less wrong by filing our theories under 'guesswork' and seeing what the actual state of play is six months from now.
Readit News
> Another method that we will leverage is pay-per-use public cloud instances. With this, anyone can spin up RHEL images in the cloud and thus obtain the source code for all packages and errata. This is the easiest for us to scale as we can do all of this through CI pipelines, spinning up cloud images to obtain the sources via DNF, and post to our Git repositories automatically.
That's quite the workaround, the rocky team has proven it's willing to get hacky if needed.
Lawyers are not going to look at this coordinated attempt to subvert a EULA and say "oh well, nothing we can do here".
Sure, they can make it more difficult by making them static, but it seems doable.
The blog post is vague on this topic and I'm not sure if you can really get all sources that way. I have my doubts but I've never tried:
https://access.redhat.com/articles/4238681
Red Hat would need to shift a few knobs and probably offend quite a few people running UBI images at least (including a zillion folks in the OpenShift community who rely on them) to cut off this current approach to getting the sources.
I wonder if Red Hat is willing to play this game of whack a mole? And IMO, was it worth it?
I suspect they're playing with unintended consequences now.
One of the nice "features" of buying CentOS, is that it meant CentOS was never going to compete - there was a clear line between community and professional, and CentOS were never going to sell you professional services.
Pushing everyone to non-RH builds has removed that line, and there's a strong chance that non-RH builds selling professional services, is going to have a higher opportunity-cost than publishing CentOS did.
And even now, technically RESF is the one producing the distro and it's also not selling professional services. Who produces the distro has no effect on who sells the services.
Could you clarify what you mean by opportunity cost here? Who faces this cost and why?
1) having something to show their customers who has the actual expertise
2) making it clear that the Red Hat of today is significantly more open than the Red Hat of 2014 when neither CentOS Stream nor UBI existed and CentOS releases were months late despite the SRPMs being on ftp.redhat.com
3) making it obvious that they are respecting the GPL, and that no one gives a flying f**k about "free as in freedom" because all the uprising was always about either the free beer or the clicks/likes.
Indeed - Red Hat is making it obvious that IBM (like most large companies) views open source as free beer - or rather free labor.
It's great when they get other people's labor for free, as long as they don't have to give away any of their own.
I feel like returning to that apparent Schelling Point could quite easily be an improvement over the Red Queen's Race that I worry is developing here.
Honestly this from this post Rocky conflicts with the "RHEL is closed source/proprietary/paywalled" narrative that people are trying to push. If RHEL was truly any of those things Rocky wouldn't have been able to continue on, but they were able to quickly find a solution, though to me it seems a bit hacky. If Rocky is pulling packages from the supposedly untested, beta of RHEL CentOS Stream, and UBI and some EC2 instance, why would I use that over something that was build cohesively in one place like Stream?
[1] https://podcast.asknoahshow.com/343 about 20 min in
The value creation is actually distributed across the ecosystem. People encounter issues, report them, and fixes are distributed. If you break that cycle and get IBM out of the loop, the process just continues elsewhere. The vast majority of that ecosystem does not pay IBM a single dollar and probably never will.
IBM is a company that is in slow decline, so the remaining Red Hat employees are facing an extended period of that company just squeezing harder and harder until nothing remains. It's death by a thousand cuts. But the bottom line is that a lot of people doing the hard work of committing actual code on behalf of the ecosystem that are currently employed there will be facing endless rounds layoffs, reorganizations, restructurings, etc. If there isn't an employee exodus happening there already, that might soon start to happen. The only question is where those people will end up.
A well funded foundation maintaining the fork of the distribution formerly known as Red Hat could be a nice destination for such people. Between Amazon, Oracle, and the countless users of Rocky, Alma, Centos, Fedora, etc. there should be plenty of brain power, motivation, and money to make that happen. They need a stable foundation. They don't need IBM to be part of that.
Don't play IBM's game on their terms. Just cut them loose. They get to do whatever they want downstream, not upstream. And they get to contribute all their fixes under GPL. That's not optional. This foundation would be free to use these fixes as they need to. Comparability with IBM's downstream distribution should not be a goal for this foundation. And if IBM wants to pretend they can do it all by themselves, they are more than welcome to try.
My prediction is that if the big supporters of this ecosystem join forces and do this, IBM will grumble a bit and then ultimately join the foundation because their alternative will be just writing off the investment they made in Red Hat and watch from the sidelines how most of the ecosystem stops depending on IBM's Red Hat.
You could have said that if they switched to using CentOS Stream, and that would even have been my favorite outcome as a Red Hat employee.
However, Rocky Linux is neither a sibling nor a fork of RHEL. It's a debranded clone that by definition cannot even have a single bugfix that isn't in RHEL. For Oracle it's okay because it's peanut money in order to annoy Red Hat, so they can afford this; for Amazon or Facebook it's no good and that's why they forked upstream at the Fedora or CentOS Stream level.
As long as Rocky Linux stays a RHEL rebuild built by a third party like the CentOS of 2010 (except backed by corporate money rather than a guy in Nebraska), Red Hat is already putting millions into "the foundation". That's what they pay for the thousand people that develop Rocky Linux, ahem RHEL. Without them, there can be no Rocky Linux at all. So, as long as Rocky's money making side keeps undermining Red Hat's money making side, game theory predicts no other outcome than death for both RHEL and Rocky.
_EDIT_: if you downvote, I'd be very glad to learn where I'm wrong
Rocky is doing something no different than what RH is doing, and if this is problematic for RH's hopes of sucking in a few $B, that's more a problem with RH's business model than it is Rocky's. They have made $Bs selling support for free software, some large part of which they didn't author, and now they want to squeeze the entire ecosystem for more.
I have zero sympathy for RH and fully support Rocky's approach here. This is a problem of RH's own creation and trying to deflect the blame onto Rocky is absurd.
Ideally, Rocky's money making side could come to some sort of agreement to share revenue with RHEL's money making side, so that in turn RHEL's software making side doesn't mind sharing code with Rocky's (lack of) software making side.
I think the abundant corporate sponsoring of Rocky Linux proves that AWS, Google, Facebook etc. are happy to pay for access to RedHat's work. They just do not agree to the way that the licensing scales (both costs and hassle).
Not necessarily. They could easily have an optional repository for "bugfixes that aren't in RHEL". Those who want bug-for-bug compatibility with RHEL for some reason could simply not enable that repository.
clang dropped to third place after Apple and Google decided to refocus on other languages, it is yet to recover from it.
FOSS is great as mantra, it turns out many people can only spend so many hours, if putting food on the table matters as existencial question.
what do you mean by this?
We see you, RedHat. You are NOT on the right path.
They sell RHEL. It's from what I gather their main source of income. Revenue from this funds things like SystemD, a lot of work in Gnome, many many things that RHEL customers and other users of Linux and desktop Linux benefit from. Of course many contributions to open source/GNU tools come from folks in no way affiliated or paid by RH and RH does use these packages but RH also provides a lot of value.
So it stands to reason, to me at least, that to allow anyone to reskin/respin/or basically just ship a RHEL clone without RH branding that is "100% bug/binary compatible with RHEL" just without the license cost is giving away something you work on for free. No rational business would allow this.
CentOS, Fedora are free. RHEL is not. Makes sense.
They've just dialled the "greed" knob a bit higher, that's all.
Exactly what the triggering incident was this time they've been very careful not to officially say (which is likely a better option than the optics of getting into a finger pointing war with a smaller target), and I suspect we won't be able to fully judge their motivations unless/until the details leak and/or are inferred by people close enough to the situation to guess correctly.
Nowadays one could probably lean on staff to manage issues and arbitrage that go-it-alone mindset over paying the RH subscription. Now that loop-hole is closed.
Arguably the specfiles are able to be copyrighted. I wonder what the license is like for those.
Highly unlikely. File hashing can be used to easily check for this.
RH can't actually do what their trying to do, but that's less important than the fact that they want to. I can't see voluntarily having anything to do with them now. I see no value in any product or service they might offer that is worth knowingly working with someone who has exposed such a lack of integrity.
Related question, as a Red Hat subscriber can I still distribute Red Hat ISO and source code? It seems like I should be able to distribute ISO images and source after obtaining them.. but not repackage it? I don't plan to impose any restrictions on the people I distribute to.
The value proposition for RHEL is ostensibly support (and a "throat to choke", for whatever that's actually worth). Red Hat's gamble is that no "legitimate" Red Hat subscriber would risk their support entitlement (and the ability to contract with Red Hat for support in the future) by exercising their rights under the GPL.
It's a clever hack. It runs counter to ideals of Free software (and I find it personally repugnant) but it's clever.
Deleted Comment
That word does not appear at all in TFA.
In the IBM/RH blog post it references[1] the word appears once, disparagingly, in the gratis sense.
I appreciate the beer / speech distinction can get tiring to explain repeatedly, but it feels like the move to distance themselves from the deeper implications & obligations of free is, shall we say, very carefully calculated.
[1] https://www.redhat.com/en/blog/red-hats-commitment-open-sour...
Though I suspect we'll both end up less wrong by filing our theories under 'guesswork' and seeing what the actual state of play is six months from now.