I sometimes look forward to outages like this just so I can read the post-downtime-resolution blog post that almost always follow. I find reading about how companies deal with issues when shit hits the fan to be really interesting.
Agreed. I almost always dial into any large scale events at my company - regardless if it concerns me directly - to listen how ppl critically evaluate data, triage errors, and determine how to mitigate the problem(s).
Whatever skill level I have in this area, I cannot nail down if it's from multiple "baptism by fire" situations, learned naturally via university courses in the scientific field, strategy video games (half joking), or other sources I cannot think of.
I've listened to some very impressive people handle serious crisis situations and I'm both in awe and curious how they achieved that level of deductive reasoning.
The same way anyone gets good at anything: practice. Knowledge of the system + experience in high pressure situations and you get pretty good at solving those problems over time.
Although it's nice that Git is Git and we can all mostly still work, it still seems foolish to rely on a single point of failure like Github. I've been toying around with the idea of creating a tool that would map the Git api to work with two+ hosting services at the same time. The effect would be something like, run "$git push" and it pushes to Github, Bitbucket, and Gitlab. I can't imagine something like this would be too difficult and would eliminate having to twiddle your thumbs while you wait for things to come back up.
I will be that one, who will remind that git server does not have to be a gitea/gogs/gitlab/onedev/pagura/git-remote-{keybase,s3,codecommit,...} - you might provide a path to the WebDav server[0][1][2], if you need a very simple git server (i.e. served internally over 192.168/16 in office network, let's say RaspberryPi with an external USB drive or temporal repository share from own laptop to your colleagues).
Is this safe? The Git docs explicitly say not to do this:
> Note that the push URL and the fetch URL, even though they can be set differently, must still refer to the same place. What you pushed to the push URL should be what you would see if you immediately fetched from the fetch URL. If you are trying to fetch from one place (e.g. your upstream) and push to another (e.g. your publishing repository), use two separate remotes.
which seems to imply that weirdness might happen if the two happen to get out of sync, or if one (specifically, the one pointing to the repository you're fetching from) fails.
For something that may be a bit safer, I believe it's possible (but haven't tested) to have multiple values for branch.whatever.pushRemote-- that should do the same thing, and has the added bonus of making the secondary remote easily fetchable.
But why not just have different remote names other than the default of “origin”? Somebody else on the thread mentioned that it might be a bit complicated to clean things up after an outage on such a “multiplexed” remote.
There's not much more to do than create a quicker remote wrapper. The Git flow here is 2+1 steps: add another remote, push to that remote (+1 is to create the remote). It would be cool to see it built into a Git plugin or wrapper!
t my last company when our internet connection went down a bunch of team members said they couldn't work because they couldn't get too GitHub. They were shocked to learn they could still collaborate by pushing their changes back and forth with other colleagues.
Perhaps what they really meant is that they couldn't get to stack overflow :-(
Github: the premier collaborative platform for FLOSS development, which itself is closed-source, made under a closed development process. Based on a distributed CVS, it has become a massive SPOF. As if that wasn't strange enough, it was acquired by Microsoft. Was that the victory of FLOSS or the defeat? How can you tell?
This falls under very specific genre of Hacker News comment that has always struck me as profoundly uninteresting.
Github built something that inarguably made development better.
- They host some huge number (tens of thousands? hundreds of thousands? millions?) of repositories for free.
- They built Github Pages for people to ship websites and host those for free.
- They built an intuitive flow for managing issues and pull requests, which is included with every repository for free.
- They integrate freely and openly with all sorts of third-party services, some of which compete directly with them. That's quite uncommon for a for-profit software platform of their size.
- They have given back immeasurably to the programming community, to the open source community, and to developers including sponsoring conferences, donating office space for events, etc.
- Their developers widely share their (i.e. done at/for Github) work and contribute upstream.
There is some argument that all of the above are self-serving for Github, which is proven false if you talk to a single developer at Github.
Do they adhere to every idealistic principle of FLOSS? No. But, honestly, who gives a crap?
So, to answer your question of WTF is going on: real, positive progress.
Except GP wasn't making any of the points you hand-waved away with "who gives a crap" (hint: all the people who complain about those things "give a crap"). They were saying that the fact we even care the GitHub went down at all is a problem with how much we've built our workflows around GitHub when Git was specifically designed to avoid this problem.
Linux kernel development doesn't shut down if kernel.org is down (heck, when kernel.org was completely pwned it only delayed the kernel release by a week or two). People still send each other emails and even if a maintainer is AWOL, other maintainers will pick patches and route them through their own trees. The only problem with the kernel development system is that it isn't easy to on-board people. But if we had a better UX for this system it would be far superior in every respect. There was a recent post by a kernel dev on how this might be improved by building on top of Secure Scuttlebutt and having a nice implementation on top[1].
* Github being closed-source doesn't really matter. Restrictive software licensing hurts the user of the software which in this case is Github Inc. and only Github Inc. As long as the protocol is open (which it is) whether hosted services that speak that protocol are open or closed source is immaterial. [1]
* Git isn't distributed. It's decentralized -- big difference. Distributed means there (usually) aren't any single points of failure. Decentralized means these are many single points of failure but failures are localized. Combine that with the reality that in most markets a "best option for most people" emerges and we get bigger and bigger points of failure.
* Microsoft has been trying to get into the socal networking game for a while now and found their match made in heaven with one based on software. Time will tell if the recent Microsoft will be a good steward -- so far it's been pretty good.
* Definitely a victory. The dominant VCS is open. More people than ever are contributing to OSS. It's easier than ever to share code.
[1] Github's on-prem offering is a completely different story and is very much a problem that it isn't OSS.
This is the perfect time to take a break. Kick back, have a coffee, contemplate your life choices. That commit can wait, that PR (i was about to merge) can wait too. It's not the end of the world.
At some point there has to be centralization. We are deploying one product, not N products for N commits. If I can't push, I can't have CI build my branch, and I can't submit a PR for review.
Because we forget how to let go of all this work stuff and get sucked into it. There is a life out there too. some one us can balance, some find it hard. this is an opportunity to just pause for a few minutes. Mentally pause.
well actually I spent the holiday weekend not working and now I have a time window to work with our east coast contractors before they stop work for the day (they don't work on the weekends or after 5PM) so having the system down isn't really end of the world but it will impact our productivity.
It's funny I thought pi-hole was somehow blocking github, but dig returned good responses.
dig raw.githubusercontent.com +noall +answer 0 < 12:54:48
; <<>> DiG 9.14.3 <<>> raw.githubusercontent.com +noall +answer
;; global options: +cmd
raw.githubusercontent.com. 6 IN CNAME github.map.fastly.net.
github.map.fastly.net. 19 IN A 151.101.20.133
Naturally the first thing I did was check here and nothing was posted. After 20-minutes of TSing I thought it was DNSSec screwing up as after disabling it everything works.
Just came here to post this. I was reading through the css-modules docs[1] and am now stuck. I can't see anything other than the main README. Anyone know of a mirror?
Readit News
Whatever skill level I have in this area, I cannot nail down if it's from multiple "baptism by fire" situations, learned naturally via university courses in the scientific field, strategy video games (half joking), or other sources I cannot think of.
I've listened to some very impressive people handle serious crisis situations and I'm both in awe and curious how they achieved that level of deductive reasoning.
see: https://git-scm.com/docs/git-remote#Documentation/git-remote...
[0]: https://cets.seas.upenn.edu/answers/git-repository.html
[1]: https://blog.osdev.org/git/2014/02/13/using-git-on-a-synolog...
[2]: https://git-scm.com/book/en/v1/Git-on-the-Server-The-Protoco...
Now I'm just annoyed that more teams I've been on haven't set this up!
> Note that the push URL and the fetch URL, even though they can be set differently, must still refer to the same place. What you pushed to the push URL should be what you would see if you immediately fetched from the fetch URL. If you are trying to fetch from one place (e.g. your upstream) and push to another (e.g. your publishing repository), use two separate remotes.
which seems to imply that weirdness might happen if the two happen to get out of sync, or if one (specifically, the one pointing to the repository you're fetching from) fails.
For something that may be a bit safer, I believe it's possible (but haven't tested) to have multiple values for branch.whatever.pushRemote-- that should do the same thing, and has the added bonus of making the secondary remote easily fetchable.
But why not just have different remote names other than the default of “origin”? Somebody else on the thread mentioned that it might be a bit complicated to clean things up after an outage on such a “multiplexed” remote.
What it does affect is the ability to do code reviews, work with issues, maybe even do releases. All the non-DVCS stuff.
Pushing to all three isn't that difficult. The hard part is reconciling after one of them suffers an outage or partition.
[1] https://stackoverflow.com/a/14290145
Deleted Comment
Deleted Comment
Perhaps what they really meant is that they couldn't get to stack overflow :-(
WTF is going on?
Mood: "Am I going crazy, or is it the world around me?" ~ https://fishbone.bandcamp.com/track/drunk-skitzo
Github built something that inarguably made development better.
- They host some huge number (tens of thousands? hundreds of thousands? millions?) of repositories for free.
- They built Github Pages for people to ship websites and host those for free.
- They built an intuitive flow for managing issues and pull requests, which is included with every repository for free.
- They integrate freely and openly with all sorts of third-party services, some of which compete directly with them. That's quite uncommon for a for-profit software platform of their size.
- They have given back immeasurably to the programming community, to the open source community, and to developers including sponsoring conferences, donating office space for events, etc.
- Their developers widely share their (i.e. done at/for Github) work and contribute upstream.
There is some argument that all of the above are self-serving for Github, which is proven false if you talk to a single developer at Github.
Do they adhere to every idealistic principle of FLOSS? No. But, honestly, who gives a crap?
So, to answer your question of WTF is going on: real, positive progress.
Linux kernel development doesn't shut down if kernel.org is down (heck, when kernel.org was completely pwned it only delayed the kernel release by a week or two). People still send each other emails and even if a maintainer is AWOL, other maintainers will pick patches and route them through their own trees. The only problem with the kernel development system is that it isn't easy to on-board people. But if we had a better UX for this system it would be far superior in every respect. There was a recent post by a kernel dev on how this might be improved by building on top of Secure Scuttlebutt and having a nice implementation on top[1].
[1]: https://people.kernel.org/monsieuricon/patches-carved-into-d...
Not sure why you had to lead with an irrelevant comment like that as the rest of your comment has interesting counter arguments.
I think Github did make development easier, but the fact that it's close source and now owned by Microsoft gives me an uneasy feeling.
With that being said, the only thing that does make it ok is the fact that Git is decentralized.
If Github was an svn host I definitely wouldn't have been ok with it or hosted anything FOSS on it.
* Git isn't distributed. It's decentralized -- big difference. Distributed means there (usually) aren't any single points of failure. Decentralized means these are many single points of failure but failures are localized. Combine that with the reality that in most markets a "best option for most people" emerges and we get bigger and bigger points of failure.
* Microsoft has been trying to get into the socal networking game for a while now and found their match made in heaven with one based on software. Time will tell if the recent Microsoft will be a good steward -- so far it's been pretty good.
* Definitely a victory. The dominant VCS is open. More people than ever are contributing to OSS. It's easier than ever to share code.
[1] Github's on-prem offering is a completely different story and is very much a problem that it isn't OSS.
Deleted Comment
Deleted Comment
And now I come back here and see this.. :D
[1]: https://github.com/css-modules/css-modules
Deleted Comment