Per the ongoing Freedesktop discussion, AWS offered to host but Freedesktop is leaning towards self-hosting on Hetzner so they can control their own destiny and sponsors can contribute cash towards the bill instead of donating hardware.
I saw their original announcement and they said that their infra (3 AMD EPYC from generations ago, 3 Intel servers from 2 generations ago, 2 80-core ARM servers) would cost $24k/month at Equinix prices. I checked Hetzner's equivalent offerings, it would be ~$1.5k/month for newer AMD servers. It would probably be even less if they went with older servers listed at their auction. And it probably would be even less if they just moved their CI runners to virtual servers on Hetzner's cloud.
Seriously, Hetzner provides so much move value per dollar, sometimes I fear that one day they will find out and just jack up the prices to match the rest.
VPS business is very different than the "cloud" space.
Yes yes there are cloud features now offered by VPS providers, but they are add ons to chase demand, they aren't positioning their offering to appeal to users wanting a comprehensive suite of services on the platform. Managed databases, SMTP as a service, deployment as a service etc etc etc. For that reasons market rates are different.
For Hetzner to bump their prices significantly they would need to build a cloud platform a la AWS/GCP/Azure. Won't happen by Xmas even if went all in. They are good at what they do and make money so they stick to that.
Hetzner also has the interesting choice of consumer-grade machines which probably work fine in cases where you are constrained by CPU power rather than memory capacity/bandwidth. You'll also lose a bit of redundancy and reliability but that might not be as big of a deal since the machines are managed by them and you can probably get things replaced quickly. For example depending on the workload the CCX43s might be replaceable by the AX52.
Meanwhile for CI runners you probably could split the big bare metal servers down into smaller individual machines and run less jobs of them. Depending on the CI load profile it might also make even more sense to scale out to the cloud on high demand as opposed to having a bunch of mostly idle machines.
Hetzner has a great price but it plays not in the same league as AWS. It's cheap and good enough for some applications but I wouldn't call Hetzner a professional service.
The WireGuard project is also in the same situation, due to Equinix Metal shutting down. If anybody would like to host us, please reach out to team at wireguard dot com. Thanks!
Saw that. Looks appealing, but I'm not particularly keen on, "We only require that you keep one sudo-enabled account on the system for us to use as needed for troubleshooting." [1] Do I want to give root access to the project's master git server to somebody I've never met, who is probably a good & nice person, but not really directly associated with the project? In general, I'm wary of places with relaxed enough informal policies that somebody could just walk over to a machine and fiddle with it. It's not that I actually intend to do some kind of top secret computing on Internet-facing machines like those, but I also don't want to have to be _as_ concerned about those edge cases when I'm deciding which things to run or host on it.
Thank you for wireguard - it's been a hugely impactful piece of software.
Do you think it would be helpful to outline what hardware resources you would need to successfully migrate the project and all the CI/CD computations to a new home? This would help people determine if they can help with hosting.
No, it's considerably more involved than that. For example, there's extensive CI: https://www.wireguard.com/build-status/ This thing builds a fresh kernel and boots it for every commit for a bunch of systems. And there's also a lot of long running fuzzing and SAT solving and all sorts of other heavy computation happening during different aspects of development. Development is a bit more than just pushing some code up to Github and hoping for the best.
Oregon State University's Open Source Lab (https://osuosl.org/) offers managed and unmanaged hosting to open source projects. They even have IBM Z and POWER10 hosting if you're into that sort of thing.
This has been brought up with freedesktop and they handwaved it away. They claim they want to DIY with donation money but they don't have a donation mechanism and I suspect they don't know how much work just handling money is.
> This has been brought up with freedesktop and they handwaved it away. They claim they want to DIY with donation money
You are making all of this sound way more definitive than the ticket. The donation money approach is the personal opinion of the sysadmin. The OSUOSL is brought up, some explanations are added that make it more attractive and remove some doubts, and beyond that it's waiting for the board to decide what's next.
Is colocation knowledge lost now? Do people no longer know how to configure a server or three, bring them to colo and run them? I don't understand how this is a story worthy of an Ars Technica article. Where's the issue?
If the issue is cost, slightly older Epyc hardware is quite affordable, and colo deals can be found for extremely reasonable costs. If it's expertise, then all they have to do is ask.
I'm sure this isn't relevant everywhere, but all my old colo hotspots within driving distance have started charging exorbitant $$ for egress, just like the cloud.
Still more economical than cloud, but it seems like this has become far too common.
Storage is MUCH cheaper when you colo, and bandwidth requirements are a large part of why you colocate instead of just running servers out of an office building that has at least two upstream connections.
I'm really curious what you think they're using now. Certainly you read the article... It says they're using bare metal servers. That's basically colo where the provider owns, but doesn't control, the hardware.
> But as we all know, RHEL/IBM only wants to take free labor and not really give back these days :(
Ludicrous. Red Hat and IBM are far from perfect, but they are absolute heroes for open source. Listing all the projects that Red Hat pays to develop would be very difficult because it's so long. They've even acquired proprietary companies and open sourced their products (while the product was still selling and highly useful!), something virtually nobody does.
Sure, they likely could. But then the complaint would be, "Arrg, I can't believe Freedesktop.org and Alpine are now effectively owned by Red Hat/IBM now, arrg!"
Also, Red Hat typically only "sponsors" open source projects that they have some business dependency on. Freedesktop.org might be a good candidate, but Alpine could be harder to justify. I don't know of any RH product that uses Alpine directly. (Most enterprises only have exposure to Alpine through container images.)
Redhat, Canonical, IBM, Oracle, Google, hell even Microsoft... There are a bunch of big actors in the Linux space that could and probably should be financing this. Also there's the Linux Foundation that is made for financing Linux projects.
Oh man that sucks! I wonder if we could pull Alpine into our colo, we recently upgraded to a full rack from 2U (it was cheaper than a quarter rack!) and have a ton of space. Plus all of our libvirt/KVM HVMs run Alpine.
Aside from some major examples, like most of the big tech companies funding the Linux kernel and maybe the Rust and/or Python Foundations in decent numbers, for the most part, corporations don't pay for open-source. That's why they love it so much: it costs ~$0, but generates immense business value for them (in that they don't have to write, debug, or maintain any of that, often essential, code or infra).
I can think of maybe three exceptions my entire career, and none of them were especially huge contributions.
Broader question, but whatever happened to every university with a CS department hosting mirrors of popular distros? I always assumed CDNs replaced them, but seeing this, maybe they didn't.
Maybe not every university, but plenty of distro mirrors are still hosted by universities, both in the US and internationally. Another example is Oregon State University mentioned elsewhere in this thread that still provides hosting + CI services; eg postmarketOS recently moved from gitlab.com to a self-hosted GitLab on OSU-provided and -hosted hardware.
Readit News
> https://gitlab.freedesktop.org/freedesktop/freedesktop/-/iss...
Seriously, Hetzner provides so much move value per dollar, sometimes I fear that one day they will find out and just jack up the prices to match the rest.
Yes yes there are cloud features now offered by VPS providers, but they are add ons to chase demand, they aren't positioning their offering to appeal to users wanting a comprehensive suite of services on the platform. Managed databases, SMTP as a service, deployment as a service etc etc etc. For that reasons market rates are different.
For Hetzner to bump their prices significantly they would need to build a cloud platform a la AWS/GCP/Azure. Won't happen by Xmas even if went all in. They are good at what they do and make money so they stick to that.
Deleted Comment
Meanwhile for CI runners you probably could split the big bare metal servers down into smaller individual machines and run less jobs of them. Depending on the CI load profile it might also make even more sense to scale out to the cloud on high demand as opposed to having a bunch of mostly idle machines.
They host quite a few open source projects there. And seem to be one of the few that also hosts for ARM and POWERPC projects.
[1] https://osuosl.org/services/hosting/details/
Hope you find a host soon!
Thank you for wireguard - it's been a hugely impactful piece of software.
Do you think it would be helpful to outline what hardware resources you would need to successfully migrate the project and all the CI/CD computations to a new home? This would help people determine if they can help with hosting.
The code is small and integrated into the kernel at this point.
Aren't your needs primarily for distributing Windows/Mac packages at this point?
You are making all of this sound way more definitive than the ticket. The donation money approach is the personal opinion of the sysadmin. The OSUOSL is brought up, some explanations are added that make it more attractive and remove some doubts, and beyond that it's waiting for the board to decide what's next.
https://osuosl.org/services/hosting/details/
If the issue is cost, slightly older Epyc hardware is quite affordable, and colo deals can be found for extremely reasonable costs. If it's expertise, then all they have to do is ask.
Still more economical than cloud, but it seems like this has become far too common.
Deleted Comment
Storage is MUCH cheaper when you colo, and bandwidth requirements are a large part of why you colocate instead of just running servers out of an office building that has at least two upstream connections.
I'm really curious what you think they're using now. Certainly you read the article... It says they're using bare metal servers. That's basically colo where the provider owns, but doesn't control, the hardware.
RHEL benefits from freedesktop and X, and as a show of good faith they could support Alpine too.
But as we all know, RHEL/IBM only wants to take free labor and not really give back these days :(
Ludicrous. Red Hat and IBM are far from perfect, but they are absolute heroes for open source. Listing all the projects that Red Hat pays to develop would be very difficult because it's so long. They've even acquired proprietary companies and open sourced their products (while the product was still selling and highly useful!), something virtually nobody does.
Also, Red Hat typically only "sponsors" open source projects that they have some business dependency on. Freedesktop.org might be a good candidate, but Alpine could be harder to justify. I don't know of any RH product that uses Alpine directly. (Most enterprises only have exposure to Alpine through container images.)
Alpine Linux: https://opencollective.com/alpinelinux
Freedesktop [edit]: ..no crowdsource option at the moment
I can think of maybe three exceptions my entire career, and none of them were especially huge contributions.
Now go ask your employer to donate.
I'm not sure how many mirrors are run by the university directly, though AFAIK MIT and RIT host theirs directly.