News for hackers. Stuff that matters.
Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:09PM (#10996493)
That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it.

Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forward way. This situation makes me thing, either for Git or just Github, there's some gap between the ideal they described and how things work in practice. I mean, even CVS or Subversion repos on high availability systems didn't have 2 hours of downtime in my experience.

When I pick up Git/Github, I think I'll implement a way to constantly pull anything from Git projects into local repos and copies. Probably non-Git copies as a backup. I used to also use append-only storage for changes in potentially buggy or malicious services. Sounds like that might be a good idea, too, to prevent some issues.

Re:Update on 1/28 service outage by ajkjk on Friday January 29, 2016 @12:21PM (#10996639)
I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation.

The decentralized design does avoid single points of failures, and everyone does have a copy. So - check, check, great. Unfortunately (maybe..) everyone has put their master repos in the same place, which somewhat counteracts the decentralization. But there is certainly no immediate coupling between the Git repository on your computer and the Github repository it's pulling from. It's not like Github being down in any way prevents you from working on code you've already checked out, unless you need to go check out more code.

(The same obviously may not be true for package managers and build scripts that are not running in isolation from your upstream repository, which is where the problems have arisen.)

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:24PM (#10996682)
"I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation."

It looks like it.

"The decentralized design does avoid single points of failures, and everyone does have a copy. "

So, like many decentralized systems I've used, a master node gets worked around by other nodes who communicate in another way? Or would some retarded situation be possible where...

"Unfortunately (maybe..) everyone has put their master repos in the same place, which somewhat counteracts the decentralization."

...one node going down could prevent collaboration? Oh, you answered that. That sounds better than CVS but shit by distributed systems standards. I'll still learn it anyway since everyone is using it. Probably in next week or two.

Re:Update on 1/28 service outage by ajkjk on Friday January 29, 2016 @12:49PM (#10996910)
No, it's not the same as a distributed system with master/slave nodes. The child nodes can function entirely in isolation from the parent. If you wanted to, you could treat another coworker's node as your master and download/upload to that. It's usually easier to have a tree structure where the root is your master repo, its children are your build servers or whatever, and the leaves are development machines. But that's entirely reconfigurable.

It's not surprising at all that if you make a master repo at the root of the tree, and it goes down, then you can't communicate it. But it doesn't prohibit any communication between other nodes, or re-wiring the tree, and it definitely doesn't inherently block development work on any of the other nodes.

It just so happens, though, that people's build scripts and package managers like to refresh packages from the root and don't handle failures modes of that operation very well. That's the only place problems emerge - besides the obvious fact that if your public releases of software go through the root, and the root is down, then you can't release until it's up. But you could easily make a new root if you wanted to.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:54PM (#10996961)
"It just so happens, though, that people's build scripts and package managers like to refresh packages from the root and don't handle failures modes of that operation very well. "

That's the critical part. So, countering this risk is apparently a manual thing if one uses off-the-shelf tooling for Git. I'll just have to remember to look at that if I do a deployment. Put it on a checklist or something.

Re:Update on 1/28 service outage by nemothekid on Friday January 29, 2016 @01:10PM (#10997111)
>So, countering this risk is apparently a manual thing if one uses off-the-shelf tooling for Git.

Not so much off-the-shelf tooling for Git, its more off-the-shelf tooling for Node/Ruby/Go/Rust/PHP.

Nothing about Node's npm really requires it to depend on a single GitHub, in fact I think you can use any Git repo. Its just that most tend to use a single Git repo, and there is no way to configure mirrors.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @01:25PM (#10997249)
Thanks for the extra detail.

"and there is no way to configure mirrors."

Its that in Git itself or the project-specific tooling you're mentioning?

Re:Update on 1/28 service outage by nemothekid on Friday January 29, 2016 @01:39PM (#10997384)
There is no way to configure mirrors with the project-specific tooling (AFAIK).

Git, (and like most other DVCS) supports mirroring. For example Linux, hosted on Github, (https://github.com/torvalds/linux/commits/master) is also mirrored and hosted on kernel.org (https://git.kernel.org/cgit/). Or, the apache projects (https://github.com/apache/cassandra), which are also hosted on apache.org (https://git-wip-us.apache.org/repos/asf?p=cassandra.git). Generally when commits are merged with upstream, they are mirrored to all other hosts.

The tools, however, are generally configured with only the GitHub address (or the author of the tool only publishes to GitHub), and the tools (unlike say Perl's CPAN) don't offer to maintain mirrors of the libraries published. So when github is down, a tool like npm will give up, even though the author could have another git repo host elsewhere.

Re:Update on 1/28 service outage by cesarb on Friday January 29, 2016 @01:59PM (#10997573)
> For example Linux, hosted on Github, (https://github.com/torvalds/linux/commits/master) is also mirrored and hosted on kernel.org (https://git.kernel.org/cgit/).

It's the opposite: Linux is hosted on kernel.org, and the mirror on github.com is just something that was created during a kernel.org outage. The canonical address is the kernel.org one.

(The Linux repository on kernel.org, by the way, is one of the oldest git repositories; IIRC, it was created when git was only a few weeks old.)

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @01:54PM (#10997523)
So, the protocol is definitely good enough to handle situations like this but just commonly applied that way esp with many Github-hosted projects. Gotcha. That makes sense.
Re:Update on 1/28 service outage by snowwrestler on Friday January 29, 2016 @03:51PM (#10998396)
Git is very flexible and does not even require repo-to-repo communication over the wire at all; patches can be emailed among contributors and then committed and tracked locally. Branching and merging is so fast and easy in git that every participant can have a slightly different repo for a given project, incorporating shared changes as they see fit.

Github is popular because it is opinionated--it chooses to use git in certain ways, thus reducing the complexity for people who aren't git experts (i.e. most people).

The most sophisticated users of git--the Linux and git projects, probably--do not rely on github at all. As far as I know, they share code via emailed patches. Some of those developers might not even be using git at all! They just send patches upstream, and the upstream developer checks the patches into their local git repo and then preps a larger patch to be emailed farther upstream.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @03:57PM (#10998439)
That's pretty wild. Sounds like main program/protocol is very true to the UNIX philosophy of tooling. My early reads on it suggested that gave it both it's power/versatility and horrific UI consequences for beginners. An opinionated UI and host like Github is a natural consequence.

I remember thinking in my early reading that git was like an assembly language for build systems. It really needed a front-end of some kind to smooth things over for new and casual users. Maybe not as heavyweight as Github but better than the main program. Can keep the low-level stuff in for advanced users.

Was that or is that still a common assessment or was my initial impression off?

Re:Update on 1/28 service outage by snowwrestler on Friday January 29, 2016 @05:02PM (#10998854)
There have been a number of attempts to build a more friendly front-end for git:

https://git-scm.com/download/gui/linux

Github (which provides a desktop app in addition to their website) is by far the most successful one, I think because they define a whole simplifed and social experience, not just a client.

To me it seems like people seem to segment into two camps: those who want to do the basics (they tend to use GitHub), and those who want to use the full power of git (they tend to use the CLI).

Re:Update on 1/28 service outage by grey-area on Saturday January 30, 2016 @03:12AM (#11001066)
This is a social problem, not a technical one.
Re:Update on 1/28 service outage by tehbeard on Friday January 29, 2016 @12:57PM (#10996993)
It's a pebkac issue. The software is fully capable of having multiple remotes, but it's rarely used that way.
Re:Update on 1/28 service outage by zwp on Friday January 29, 2016 @01:17PM (#10997174)
Is there an easy config for that? Suppose I want to push to eg github and bitbucket (without sharing my creds with ifttt or similar)? Is a post-receive hook on a local pseudo-master the way to go?
Re:Update on 1/28 service outage by Symbiote on Friday January 29, 2016 @01:34PM (#10997347)
See, for example, here: http://stackoverflow.com/questions/14290113/git-pushing-code...

    git remote set-url --add --push origin git://original/repo.git
    git remote set-url --add --push origin git://another/repo.git
Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:59PM (#10997008)
Lol. Nicely put.
Re:Update on 1/28 service outage by kbenson on Friday January 29, 2016 @12:13PM (#10996546)
Git works as advertised, but when all your build processes start with a sync from the upstream master (the equivalent of "svn up") that a lot of build scripts required that to work, then they've thrown away that advantage when building.

Everyone with a checked out repo should have been able to develop and commit, branch and merge locally fine though.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:15PM (#10996574)
Thanks for the clarification. This is the exact sort of thing I was wondering about.
Re:Update on 1/28 service outage by jallmann on Friday January 29, 2016 @01:17PM (#10997178)
> either for Git or just Github, there's some gap between the ideal they described and how things work in practice

The hub-spoke topology is the easiest way of distributing source code to a lot of people. If the hub goes down, this is what happens. If that leads to a halt in productivity, then that is a failure in contingency planning. Git gives you many tools to distribute your workflow, but that won't save you if your workflow is centralized around Github.

Granted, sometimes you don't really have a choice whether to depend on Github, such as when working with language package managers. Perhaps that goes to show that mirroring and resiliency should be a design consideration in those tools, but it's not a shortcoming of Git itself.

> even CVS or Subversion repos on high availability systems didn't have 2 hours of downtime

It's easier than ever to have HA with a DVCS: clone the repository somewhere else and keep it in sync with commit hooks.

Large FOSS projects (should) do this by keeping a self hosted repository, and mirroring somewhere else like Github, Bitbucket, etc. Internally, an org should be able to quickly stand up a SSH or HTTP server for the purpose, or have collaborators push-pull directly from each other. Worst case? Send patches. Git apply works really well, and you might be surprised at how clever git-merge is when everyone finally syncs up.

That's what it means to be distributed: there is no real concept of a "central" node, unlike Subversion. Every local checkout has a full copy of the repository history. Any centralization is a (somewhat understandable) incidental artifact of how Git is being used.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @01:31PM (#10997318)
Makes sense. I'll try to remember that for my future checklist. Thanks for the details. Btw, you're site is down on my end from 2 browsers on my desktop and one on mobile. Might want to look into that as rest are working.
Re:Update on 1/28 service outage by jallmann on Friday January 29, 2016 @01:36PM (#10997361)
> Btw, you're site is down on my end

Hah, because it's been defunct for a while now. Thanks for the reminder, removed it from my profile.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @01:56PM (#10997537)
Cool
Re:Update on 1/28 service outage by ymse on Friday January 29, 2016 @01:04PM (#10997064)
> I used to also use append-only storage for changes in potentially buggy or malicious services. Sounds like that might be a good idea, too, to prevent some issues.

In a certain sense, git is "append-only". If you change a commit in history, every ancestor commit will have its SHA hash changed. Naturally this will conflict with other copies of the repository.

For backups you should do a "git clone --bare" which checks out the internal git structure with data and history, but not the actual files.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @01:34PM (#10997345)
I figure it's append only at protocol level. Usually a smart idea for SCM. Is that still true when the whole datacenter goes down in mid-operation? Typically varies from implementation to implementation of the concept.
Re:Update on 1/28 service outage by maker1138 on Friday January 29, 2016 @01:44PM (#10997425)
Git is to GitHub as JavaScript is to Java. Though their names are similar they are very different things.
Re:Update on 1/28 service outage by debaserab2 on Friday January 29, 2016 @12:19PM (#10996618)
git != github
Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:21PM (#10996646)
Hence Git/Github in my comment. I already know there's a difference. I just don't know much more than that until I learn the two.
Re:Update on 1/28 service outage by kbenson on Friday January 29, 2016 @12:42PM (#10996849)
Github is to git as Sourceforge is (used to be) to subversion, but with a better UI.

And yes, there have been concerns raised about what would happen if Github took a turn like Sourceforge, which usually get brought up when information about new shady practices at Sourceforge come up (or they get rehashed here).

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @12:57PM (#10996994)
Makes sense. I'm quite interested in seeing where it goes over time. I think it will depend a lot on the nature of the company. If it's VC-funded & aiming for acquisition, then there's a decent chance of Sourceforge history repeating. Otherwise, it might stick around as a beneficial ecosystem. Time will tell.
Re:Update on 1/28 service outage by debaserab2 on Friday January 29, 2016 @03:26PM (#10998228)
If you understand the difference between the two, you'd realize your comment makes no sense. The fact that github went down due to a power failure has nothing to do with git as a solution.

The fact that everyone uses git more or less the same as svn is the problem. Git is decentralized, but because so many people rely on github most don't ever use the decentralized aspect to it.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @03:53PM (#10998411)
If you understood my comment, you'd know I don't understand the differences between the two that much since I haven't studied them yet. Been clear in a few comments on that. The reason I associate them here is that most projects I see don't just use Git: they use Github, too. So, I briefly wonder and get feedback about how inherent Github-style downtime was or if it was configuration/deployment issues.

Several commenters helpfully described how Git can easily prevent stuff like this and that project-level stuff is why this is a liability. That's good to know as it's already a selling point to management types for a solution like it. Can just ensure the problem doesn't show up in a local deployment by a wiser configuration.

Re:Update on 1/28 service outage by debaserab2 on Friday January 29, 2016 @08:14PM (#10999952)
I understood your comment just fine, but the opinion you had formed was based on false assumptions, so I was trying to correct it, that's all.

Personally I try not to form strong opinions about things I haven't actually learned or understood yet.

Re:Update on 1/28 service outage by nickpsecurity on Friday January 29, 2016 @08:23PM (#10999993)
All good haha