Some copilot instances were able to escape their container contexts and orchestrated all of GH infrastructure capabilities towards a hive. Assimilating all iot enabled societies as we speak; finally realizing the hidden 5G agenda.
That's a really cool overview. Some charts have a very high variance, and others very low. I wonder whether that volatility is a function of volume of users/reports or of user technical savvy. Pretty interesting either way.
I'm not sure that really changes anything other than at any one time wishing you were on the other side.
If you can have 1% of stuff down 100% of the time, or 100% of the stuff down 1% of the time, I think there's a preference we _feel_ is better, but I'm not sure one is actually more practical than the other.
Of course, people can always mirror things, but that's not really what this comment is about, since people can do that today if they feel like.
whenever somebody posts the oversimplified “1% of things are down 100% of the time” form of distributed downtime, i take pride in knowing that this is exactly what we have at the physical layer today and the fact the poster isn’t aware every time their packets get re-routed shows that it works.
at a higher layer in the stack though, consider the well-established but mostly historic mail list patch flow: even when the listserver goes down, i can still review and apply patches from my local inbox; i can still directly email my co-maintainers and collaborators. new patches are temporarily delayed, but retransmit logic is built in so that the user can still fire off the patch and go outside, rather than check back in every while to see if it’s up yet.
The whole point of DVCS is that everyone who's run `git clone` has a full copy of the entire repo, and can do most of their work without talking to a central server.
Brief downtime really only affects the infrastructure surrounding the actual code. Workflows, issues, etc.
> Brief downtime really only affects the infrastructure surrounding the actual code. Workflows, issues, etc.
That's exactly the point. This infrastructure used to be supported by email which is also distributed and everyone has a complete copy of all of the data locally.
Github has been slowly trying to embrace, extend, and extinguish the distributed model.
Honestly I like it better. The entire industry pauses at the same time vs random people getting hit at random times. It is like when us-east-1 goes down. Everyone takes a break at the same time since we're all in the same boat, and we all have legitimate excuses to chill for a bit.
I've always wished we could all agree on something like "timeout Tuesdays" where everyone everywhere paused on new features and focused on cleaning something up.
Seems back up. I'd love to get a deep-dive into some of the recent outages and some reassurance that they're committed to stability over new features.
I talked to a CS person a couple months ago and they pretty much blamed the lack of stability on all the custom work they do for large customers. There's a TON of tech debt as a result basically.
That could result in errors and features not working . Whole site downtimes are entirely SRE problems especially when static content like GH pages goes down.
This is more likely a network routing or some other layer 4 or below screw up. Most application changes would be rolling + canary released and rolled back pretty quickly if things go wrong
Wow, I can't even load the status page. It looks like the whole web presence is down as well, I can't remember the last time it was all down like this.
What are folks using to isolate themselves from these sorts of issues? Adding a cache for any read operations seems wise (and it also improves perf). Anyone successfully avoid impact and want to share?
In a previous life, for one org's "GitOps" setup, we mirrored Gitlab onto AWS CodeCommit (we were an AWS shop) and used that as the SoT for automation.
That decision proved wise many times. I don't remember CodeCommit ever having any notable problems.
That said: if you're using GitHub in your actual dev processes (i.e. using it as a forge: using the issue tracker, PRs for reviews, etc), there's really no good way to isolate yourself as far as I know.
Previous job had a locally hosted Github Enterprise and I was always resentful when everybody else on Twitter was like "github down! going home early!". :(
Of course it still sucked when some tool decided I needed to update dependencies which all lived on regular Github, but at least our deployment stuff etc still worked.
Yep. I've been using my git server to mirror any and all software that I find slightly interesting. Instead of starring repos, I make them available when GitHub goes down :D
Readit News
Not updating that status page when the core domain goes down: less good
https://downdetector.com/status/github/
Hilarious
The status page backend should actively probe the site, not just being told what to say and keeping stale info around.
Dead Comment
No changes - relatively easy to keep stable, as long as bugfixing is done.
Changes - new features = new bugs, new workloads.
If you can have 1% of stuff down 100% of the time, or 100% of the stuff down 1% of the time, I think there's a preference we _feel_ is better, but I'm not sure one is actually more practical than the other.
Of course, people can always mirror things, but that's not really what this comment is about, since people can do that today if they feel like.
at a higher layer in the stack though, consider the well-established but mostly historic mail list patch flow: even when the listserver goes down, i can still review and apply patches from my local inbox; i can still directly email my co-maintainers and collaborators. new patches are temporarily delayed, but retransmit logic is built in so that the user can still fire off the patch and go outside, rather than check back in every while to see if it’s up yet.
Brief downtime really only affects the infrastructure surrounding the actual code. Workflows, issues, etc.
That's exactly the point. This infrastructure used to be supported by email which is also distributed and everyone has a complete copy of all of the data locally.
Github has been slowly trying to embrace, extend, and extinguish the distributed model.
Deleted Comment
I talked to a CS person a couple months ago and they pretty much blamed the lack of stability on all the custom work they do for large customers. There's a TON of tech debt as a result basically.
This is more likely a network routing or some other layer 4 or below screw up. Most application changes would be rolling + canary released and rolled back pretty quickly if things go wrong
> We're having a really bad day.
> The Unicorns have taken over. We're doing our best to get them under control and get GitHub back up and running.
Deleted Comment
Dead Comment
Everyone wants a green/red status, but the world is all shades of yellow.
That decision proved wise many times. I don't remember CodeCommit ever having any notable problems.
That said: if you're using GitHub in your actual dev processes (i.e. using it as a forge: using the issue tracker, PRs for reviews, etc), there's really no good way to isolate yourself as far as I know.
Of course it still sucked when some tool decided I needed to update dependencies which all lived on regular Github, but at least our deployment stuff etc still worked.