Readit News logoReadit News
codeflo · 2 months ago
My theory: Some manager's KPI is to increase the number of sold GitHub runner minutes. So they did some market research -- not enough to have a clear picture, but barely enough to be dangerous -- and found that some companies use self-hosted runners for cost reasons. So they deploy a two-pronged strategy: lower the cost of GitHub runners, and charge for the use of self-hosted runners, to incentivize switching.

This fails for several reasons that someone who actually uses the product might have intuited:

(a) For some use-cases, you can't switch to GitHub's runners. For us, it's a no-go for anything that touches our infrastructure.

(b) Switching CI providers isn't hard, we had to do it twice already. Granted, most of our CI logic is in a custom build script that you can run locally, and not in the proprietary YAML file. But to be honest, I'd recommend that sort of setup for any CI provider, as you always want the ability to debug things locally.

(c) GitHub Actions doesn't get the amount of love you'd expect from something billed as a "premium service". In fact, it often feels quite abandoned, barely kept working. Who knows what they're brewing internally, but they didn't coordinate this with a major feature announcement, and didn't rush to announce anything now that they got backlash, which leads me to believe they don't have anything major planned.

(d) Paying someone -- by the minute, no less -- to use my own infrastructure feels strange and greedy. GitHub has always had per-user pricing, which feels fair and predictable. If for some reason they need more money, they can always increase that price. The fact that they didn't do that leads me to believe this wasn't about cost per se. Hence the KPI theory I mentioned above: this wasn't well-coordinated with any bigger strategy.

benterix · 2 months ago
> Switching CI providers isn't hard, we had to do it twice already. Granted, most of our CI logic is in a custom build script that you can run locally, and not in the proprietary YAML file. But to be honest, I'd recommend that sort of setup for any CI provider, as you always want the ability to debug things locally.

I believe this has been a CI/CD best practice for over a decade. Even in venerable Jenkins, this is one of the core principles when designing pipelines[0]: don't give in to the temptation to do fancy Groovy stuff, just use simple shell commands in steps, and you will be grateful to yourself several times years later.

[0] https://www.jenkins.io/doc/book/pipeline/pipeline-best-pract...

freedomben · 2 months ago
It has been best practice for over a decade, but for reasons I don't understand, nearly every developer I've worked with just wants to do the lock-in/propietary route and is entirely unpersuaded by the "portability" argument. I've now seen it burn teams hard multiple times now. At that point people realize the wisdom in the external scripts, but then a new wave of devs come in and start the whole cycle over.
markus_zhang · 2 months ago
I don’t know why, but the linked page only shows the table of contents on iPhone Safari, but whence I switch to reader mode it shows the actual best practices. Anyway thanks for sharing!
g947o · 2 months ago
https://news.ycombinator.com/item?id=46189692 from a few days ago pretty much tells me that any company that slightly cares about security cannot possibly depend on GitHub runner for their CI (except maybe the smallest/simplest projects). It is just one compromised package away from ruining everything.
altmanaltman · 2 months ago
"Hey ChatGPT, how do I increase the number of GitHub runner minutes? DO NOT suggested anything illegal, research hard"
baobun · 2 months ago
(e): Community software like act and Forgejo/Gitea Actions have made it a lot easier to run GitHub Actions workflows without involving GitHub and are decreasing the friction of migration.
dirkc · 2 months ago
I agree with (c) - I can't quite pinpoint it, but I've had that feeling myself several times.

Deleted Comment

bilekas · 2 months ago
> GitHub stated that it has canceled the price increase after reviewing developer feedback. It added that it will take time to listen to customers and partners.

I get the feeling they got the feedback that their runners are not as indispensable to developers as they thought and realized they would lose a significant amount of users. Now if only they would listen to the feedback about windows 11 and their forced copilot we might be onto something.

embedding-shape · 2 months ago
I wonder if Microsoft will ever get that asking users before making changes can help them avoid looking bad in public.

Maybe half of all clients I work with use GitHub Actions for CI (the rest basically all use Jenkins), most of those using Actions use self-hosted runners for performance and security reasons, almost all of them reached out to me asking how difficult it would be to move away from GitHub Actions yet continue using GitHub.

Do you think these companies now suddenly stop wanting to move away from GitHub Actions because Microsoft suddenly changed their mind? I don't think so, probably less priority, but it will happen, because now the cat is out of the bag.

If they'd just do user research before announcing changes and not use announcements as "testing the waters", I'm sure they'd see a lot less churn. But I guess some number counting team somewhere in Microsoft figured out they'd make more money by charging people to run software on their own hardware, so maybe I'm just dreaming.

Xylakant · 2 months ago
Fun thing is that almost every other CI as a service provider charges you in some shape or form for self hosted runners. CircleCI limits the number of self-hosted Job Running in parallel based on your plan and charges a fixed base fee per seat.

So moving away from GHA will not make self-hosted runners free, they’ll move into a different pricing structure that may or may not be beneficial.

And I think charging for self-hosted runners is actually fine. They’re not free for the provider either - log aggregation, caching of artifacts, runner scheduling, implementing the runner software etc are non-trivial problems for any larger CI system.

So I’m actually fine with the proposed change since it also gives me the power as a customer to say “hey, I’m paying for this, fix it.”

renegade-otter · 2 months ago
For a small company, the things that MS is doing these days would be fatal. They can screw up time after time and just move on, with the executives getting bonuses for their Wharton School of Business genius.

That said, they are burning that capital really quickly.

markus_zhang · 2 months ago
MSFT used to be extremely good about that in the 90s. From the book Showstoppers: MSFT sent not one, but three batches of NT 3.1 beta to external developers before the final release.

The idea nowadays is iterate fast and break things (as long as it’s not your wallet or your leg).

anothernewdude · 2 months ago
I've already jumped ship. Switching source control host was actually pretty easy. Builds still working just fine.
thomasnno · 2 months ago
Great! Who did you jump to ?
renegade-otter · 2 months ago
In the age of LLMs, "rewrite these actions to use XYZ instead" also makes this easier than ever. Not a one-shot solution, probably, but most likely not as challenging.

Microsoft thinks it has a moat by locking people in, but they ticked me off hard enough (finally) for me to switch my gaming Alienware rig to Mint. It starts faster, too.

grub5000 · 2 months ago
> Now if only they would listen to the feedback about windows 11 and their forced copilot we might be onto something.

You can just uninstall Copilot? It’s nowhere on my Surface Laptop 7 with W11.

bilekas · 2 months ago
It has reappeared on mine after mandatory windows updates which is frustrating and also it looks like it will be arriving on my TV soon too without the option to remove it.

> https://www.tomshardware.com/service-providers/tv-providers/...

So it's not a stretch to assume they will continue to force it in their OS.

renegade-otter · 2 months ago
Microsoft will remove something after an outcry and then will later get it back with no option to remove. This happens all the time. People have no attention spans these day as they move to the new outrage hashtag of the day, so this works.
evilduck · 2 months ago
Why should anyone have to take action against it? Good products don't need to be forced upon users, an obnoxious ad in one of the dozen places Windows shows advertising would have sufficed. People even willingly fork over cash for ChatGPT and Claude and those don't even have OS ad placements or forced installs.
newdee · 2 months ago
For now
nerdjon · 2 months ago
That would be my guess, I know personally yesterday I finally setup Forgejo and today I plan to evaluate its runners or even just using a dedicated CI like woodpecker.

Not fully sure what I will do regarding any open source repo's yet, but at least anything private I am already in the process of moving away.

This was something I already wanted to do for privacy concerns (especially possibility using private repo's to train AI) so this was just the push I needed.

klaussilveira · 2 months ago
We jumped ship too. Forgejo has been amazing.
p_j_w · 2 months ago
> Now if only they would listen to the feedback about windows 11 and their forced copilot

I think they'll take the opposite lesson. Copilot hasn’t lost them many users because Windows users are locked into the ecosystem and unable to leave. They will try to get GitHub into a position similar to that and then try this shit again.

amluto · 2 months ago
I feel like I could specify and vibe-code a CI workflow system that would be dramatically better (for a single organization’s workflow) than GitHub Actions. And hosting it would be barely more complex than hosting a GitHub Actions self-hosted runner.

The stack would be:

Postgres, as a job queue and job status tracker. The entire control plane state lives in here. Even in a fairly large org, the transaction rate would be very, very low.

An ingestion agent. Monitors the repository for pushes and PRs.

A job agent. This runs a in a sandbox and gets the inputs from GitHub and runs what is effectively a workflow step. It doesn’t get any secrets — everything it wants to do is either accomplished in the form of JSON output, blob output, or an org-specific API for doing things that don’t fit the JSON output model.

A thing to handle results. This is a simple service, connected to the database, that consumes the JSON job results and does whatever is needed (which would mostly consist of commenting on PRs or updating a CI status dashboard). For CD workflows, the build artifacts would be sent to whatever registry they go to.

A configuration system, which would be some files somewhere, maybe in a git repository that is not the repository that CI is being done on. (GitHub’s model of Actions config being in-band in the repository is IMO entirely wrong.)

And that’s about it.

I’m not suggesting that I could duplicate the GitHub Actions in a weekend. But I wouldn’t want to. This would be single-tenant, and it would support exactly the features that the organization actually uses. Heck, even par-for-the-course things like SSO aren’t needed because the entire system would have no users per se :)

bilekas · 2 months ago
I'm not sure why this is so downvoted given the climate.

I guess it's the AI driven approach. These things, critical infra, are always done better with a few eyes in it.. introducing irresponsible ideas of "I'll just remake it with Claude without knowing the underlying infra" can hit a few nerves, also add a few lower level bugs, exploits etc.

I don't think it's fair for the downvoted but I think it's worth discussing where we draw the line.

Edit: I think AI is a tool not a replacement.

Dead Comment

tom-9999 · 2 months ago
1. Announce price increase generating bad publicity.

2. Kill bad publicity with blog pretending to be understanding and taking on feedback while "pausing" the increase.

3. Implement price increase a few months later when the bad publicity wave is over, and its old news so wont generate new headlines.

Uehreka · 2 months ago
Nah, that doesn’t work when the substance of the change is this intense and has an actual effect on peoples’ bottom lines. If they wait a few months and try again, people will see their bills go up immediately and they’ll all get mad again. I don’t know what GH will do next, but if they try to do that, it will definitely backfire.
beaker52 · 2 months ago
You might be surprised at how much you’re willing to surrender if someone gave you some time to come to terms with it.

It’s just a question of giving you enough time to move on from anger/shock/fear to toward acceptance. It’s like magic and is used all the time.

> Nah, that doesn’t work when…

Sounds like it could be another well known stage of the process called denial. Denial is when you tell yourself that something isn’t possible which makes you feel safer, when in fact you’re already moving toward acceptance - acceptance that you’re going to leave, or pay the price.

lbreakjai · 2 months ago
When they eventually roll it out (And they will, they always do), everyone will have had plenty of time to run the numbers and either come up with a plan, or just swallow the pill.

If you still complain in a few months then that's on you, because you've been warned.

dijit · 2 months ago
A lot of evidence that this is not the case.

It happens a lot, Atlassian being a really prominent example with their “SaaS only” change.

Vespasian · 2 months ago
sounds like it's time to increase their vendor lock in then an make sure they are not as compatible with other solutions.

I fear this would be the obvious conclusion.

csomar · 2 months ago
No. There is real serious money involved here. Usually, the people who self-host are maxing their runners (otherwise it makes more sense to use minute billing). So this will affect them by roughly doubling their servers cost. Think if some company had a $15K/month bill in self-runners, they'll now also get a $15k/month additional bill from GitHub.

Many people will switch for that kind of money.

withinboredom · 2 months ago
We use dedicated machines for our runners. Each machine has like 16+ cpus, 64gb+ of ram. Costs are <2k per month. This pricing change would have cost more than the servers we're running on.
linuxftw · 2 months ago
This is my take as well.

Personally, I think this is all overblown anyway. Their pricing seems fair to me. Too many people are used to getting something for nothing. Most companies will just pay the new prices, because the time to develop and setup an alternative will far exceed just paying the new fees to GH (when you account for engineering cost).

redrove · 2 months ago
Sounds like Chat Control.
djeastm · 2 months ago
If this is the case then they don't really understand developers at all. We'll complain in a few months just as much as we do now.
estimator7292 · 2 months ago
They have not shown any hints that they understand developers since Ballmer
otterley · 2 months ago
This is the official message: https://x.com/github/status/2001372894882918548

"We’ve read your posts and heard your feedback.

"1. We’re postponing the announced billing change for self-hosted GitHub Actions to take time to re-evaluate our approach.

"2. We are continuing to reduce hosted-runners prices by up to 39% on January 1, 2026.

"We have real costs in running the Actions control plane. We are also making investments into self-hosted runners so they work at scale in customer environments, particularly for complex enterprise scenarios. While this context matters, we missed the mark with this change by not including more of you in our planning.

"We need to improve GitHub Actions. We’re taking more time to meet and listen closely to developers, customers, and partners to start. We’ve also opened a discussion to collect more direct feedback and will use that feedback to inform the GitHub Actions roadmap. We’re working hard to earn your trust through consistent delivery across GitHub Actions and the entire platform."

sltr · 2 months ago
Palmer's tweet said a similar thing and added, "we gave away 11.5 billion build minutes (~$184 million) to support OSS last year".

$184M in profit or cost?

It's called a loss leader, not a gift, and it's a marketing and adoption tactic. They already bought the machines which cost about as much to run idle as at 100% utilization. Might as well put that idle capex and opex to use.

Or just collectively bill OSS the $184M and stop signaling virtue.

otterley · 2 months ago
We don’t know whether GitHub has idle capacity to spare. If they’re auto scaling their resources like a well-architected workload should, then they probably don’t. Even if they’re not downscaling their compute, they may prefer to run other low-priority jobs on it.
denismi · 2 months ago
Outside of work, I'm a very sporadic coder. On some side-projects where I'm using Actions, I'll have an inspired few days of progress followed by completely idle weeks/months/quarters.

Losing free Actions doesn't particularly bother me, and I have no issue with paying what is most likely a negligible amount, but I don't really want to have a credit card on file which could be charged some unbounded amount if somebody gets into my account. I've shut down my personal AWS for similar reasons.

Is there any way of me just loading up a one-time $20? That will probably last well into 2027, and give me the peace of mind that I can just let it run. If my account's compromised, or I misconfigure something that goes wild, I am perfectly happy to write off that amount and have my incredibly-low-stakes toy projects fail to build.

robertjpayne · 2 months ago
Put a spend limit in GitHub and issue a chargeback if they ever bill you more.
embedding-shape · 2 months ago
Setup something like CircleCI that mainly relies on paid users of their main product, and has a free plan. Microsoft currently seem to be in the process of figuring out how to lower the costs of GitHub for free users, since I'm guessing they make their actual money on other segments and products.
spockz · 2 months ago
There are several “virtual credit card” providers that allow you to generate additional cards, set limit on them like amounts and who can charge the CC. The availability varies per geography.
rjmunro · 2 months ago
The problem with that is you might still get a huge bill if something goes wrong, then they try to charge it to your card at the end of the day/week/month/whatever, and it fails.

Now you still owe them the money, but haven't paid, so they tell you to pay on another card. If you refuse, they start debt collection against you and you could end up with your credit rating being affected, and maybe court cases and so on.

I want give the company an amount of money, then know that it's run out and I have to pay for more. You can set monthly limits (https://github.com/settings/billing/budgets), but if you are like me and have personal projects that you work on for a week or two a few times a year, that doesn't really work.

blitzar · 2 months ago
As someone who has had an occasional nasty AWS bill, the Ai providers using a pre-pay credit system is something I approve of, and would love to see everyone else offer.
hi_hi · 2 months ago
I just want to say I found this quite an insightful comment. I similarly would love to use a pay-as-you-go pricing model as a way of safely trying out various SaaS services.

Unfortunately I feel it is not in the SaaS businesses interests, who want to replicate the gym membership model where the 70% who don't use the service are supplementing the other 30% who use it frequently.

htrp · 2 months ago
Realistically you aren't their target market. They're targeting the enterprises who already have self hosted runners and aren't interested in switching to Actions minutes.
kevin061 · 2 months ago
"Postponing" means they will just do it later.

The writing is on the wall. Up to you if you wish to continue using and trusting Microsoft.

thomascountz · 2 months ago

   "Postponing" means they will just do it later.
For the record, planning to do something later than originally planned is the definition of "postpone." Nevertheless, coupling to any vendor is a form of technical debt, and it's always a good idea to take stock and evaluate if it's time to start repaying it.

dvh · 2 months ago
There's one thing I don't understand. Isn't GitHub action just "take a repo, do something with it, save something somewhere". So how is it different than writing a bash script that "clones the repo, do something with it, pushes the changes back"? If actions became paid feature, wouldn't that just generated myriads of show hn posts like "I recreated GitHub actions in xyz"?
benterix · 2 months ago
Yes, in theory any CI/CD (whether Github, Gitlab, Jenkins etc,) is just a shell script with some warpping. But this wrapping matters because it's convenience: how you integrate with the repo, how you deal with variables, secrets, caching, deployment security and so on. Some people roughly figured this out and proposed some ways which other people learned, so switching is always a bit of a pain in various ways. But it is definitely possible - I did quite a lot of pipeline migration in my life and this is definitely not a blocker for a project of small to medium to biggish-size.
000ooo000 · 2 months ago
Where does one get to suffer YAML hell then??
muixoozie · 2 months ago
Hand roll something on k8s. Yaml seems unavoidable in that ecosystem. :)
nijave · 2 months ago
Part of the advantage is a fully hosted service where you don't need to worry about a control plane managing and scheduling jobs.

There's some cost involved there--especially if you're offering hosted runners (you need some capital to buy infra to run jobs)

There's also advantage of limiting how many companies you share your code with. If you're using GitHub, you're already trusting them with your source code so GitHub Actions doesn't require more trust or an additional entity to manage/pay

baobun · 2 months ago
This is likely the reason behind the recent push of "Trusted Publishing" from NPM. They are trying to make people consider GitHub (and GitLab) in its own higher tier with regards to supply-chain security by decree.

If you rely on "Trusted Publishing" you are assisting Microsoft in making a moat for their CI platform.

Use cryptographic signatures, not implicit trust in a hosted platform.

ClikeX · 2 months ago
Yes, and there are several tools that do just that already. The thing Github Actions offers is logging and storage of artifacts.

This change is a very weird one. Because the teams that are capable of setting up their own runners have the means to easily jump ship to another CI platform and skip these fees altogether.

franktankbank · 2 months ago
Yea but a bash script isn't declarative so thats somehow bad.

(DECLARATIVE CI WHY?????)

000ooo000 · 2 months ago
Straightforward statements with a few conditionals considered harmful. Here learn this grotesque vendor-specific DSL hamfisted into YAML instead.

Deleted Comment

CafeRacer · 2 months ago
It's like bash scripts, but with ability to debug them.
Kinrany · 2 months ago
There are few things less debuggable than Github Actions. Bash scripts isn't one of them
IshKebab · 2 months ago
> Although we gave away 11.5 billion build minutes (~$184 million) to support OSS last year

Interesting, I was trying to estimate how much they spent on free actions per year. I thought it would be around $100m. This is the first actual number I've seen.

I expect the $184 million figure is the sale price rather than the actual cost to GitHub, and given that competitors offer the same service for 3-10x less it's probably more like $80m overall I'd guess.

Still a pretty huge amount of money that I don't think any competitors can really hope to match.

blibble · 2 months ago
I wonder how much of that $80 million is garbage code like safe_sleep.sh