News for hackers. Stuff that matters.
Re:GitHub: October 21 Incident Report by _verandaguy on Monday October 22, 2018 @10:03AM (#18274701)
I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod.

It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks by nation-state-level actors because of their role as a code hosting platform.

EDIT: GH hasn't been targeted regularly, but it has been so historically, so this is a plausible thing which might happen again.

Re:GitHub: October 21 Incident Report by Klathmon on Monday October 22, 2018 @10:30AM (#18274937)
It's a tradeoff, just like anything else.

We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it.

Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.

Re:GitHub: October 21 Incident Report by bschwindHN on Monday October 22, 2018 @08:59PM (#18279965)
> and is probably more likely to go down.

People often say this but I wonder if it's true. Your replication wouldn't be handling the same load as GitHub itself, so perhaps the issues that GitHub experiences would rarely happen to a self hosted version.

Re:GitHub: October 21 Incident Report by colonelpopcorn on Monday October 22, 2018 @11:23AM (#18275366)
On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.
Re:GitHub: October 21 Incident Report by elliotlarson on Monday October 22, 2018 @11:30AM (#18275449)
But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.
Re:GitHub: October 21 Incident Report by module0000 on Monday October 22, 2018 @11:50AM (#18275653)
Well, your code is your companies' IP. It might be prudent to have that IP on-prem(eg Gitlab). Every business has varying requirements, but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror. Disclaimer: my employment has been at megacorps so far, smaller shops may not do this.
Re:GitHub: October 21 Incident Report by Klathmon on Monday October 22, 2018 @11:58AM (#18275746)
with git, you have a mirror on every single developer machine (kinda... depending on what you consider "all" of the code).

We also have a full copy on our CI server (which is hosted on another service, so still not "in-house"), and we have a copy of at least the master branch (and all it's history) on our production boxes (which also gets pulled into our whole backup system there).

In a disaster recovery scenario, that's more than enough for me.

Sure, if github blinked out of existence we would probably be at a fraction of our normal productivity for a while until we fully recover everything and find new workflows, but the risk vs reward there is well within the margins of what I'd consider acceptable for a company like us.

Re:GitHub: October 21 Incident Report by ams6110 on Monday October 22, 2018 @12:06PM (#18275842)
> In a disaster recovery scenario

That's not really enough. You need more than "all the code is around here somewhere" you need a plan with specific steps that have been tested.

Re:GitHub: October 21 Incident Report by twunde on Monday October 22, 2018 @12:43PM (#18276167)
Honestly for git, it probably is enough. We're talking about someone deleting your github account or github closing overnight with no warning (it's been acquired by Microsoft, so it's much more likely that the company you're working for will shutter). It should take ~30 minutes to push your repo to another provider including looking up instructions. Unlike database backups, there is rarely any data loss and any data loss should be recoverable. It's also not client-facing, but is a temporary problem similar to wifi going down at your office. An inconvenience and hassle, yes. Long-term problem, no.

Furthermore, the problem with these disaster scenarios are that there are much more dangerous problems than your account being deleted. Someone with admin access, could insert a back door or sell your source code to someone else. That's honestly scarier.

Re:GitHub: October 21 Incident Report by CamTin on Monday October 22, 2018 @01:25PM (#18276505)
That's probably the case for valley-style startups where the whole team can fit in a room and they all hack on the same handful of repos, but most "enterprise" customers will have hundreds of repos with not necessarily anybody hacking on most of them at any given moment. It's very good policy for such organizations to have a plan in place to "break glass in case Github is down" with local mirroring of all data and a tested process for doing deploys without Github.
Re:GitHub: October 21 Incident Report by module0000 on Monday October 22, 2018 @03:13PM (#18277418)
We have exactly that... in our DR plan, there is a section for how to cope with the 3rd party source control provider being unavailable/compromised/etc. Update DNS for the equivalent of "upstream-git.foo.com" to an internal address, and continue business as usual.

It's like you said, smaller shops probably think it's over-planning and overkill, but we do indeed have 100's of projects that are "mission critical", that might not have been touched in 1+ years.

Re:GitHub: October 21 Incident Report by mk89 on Monday October 22, 2018 @12:04PM (#18275824)
>but I've yet to be employed at one using 3rd party hosting for source control without an on-site mirror.

For me it's actually the opposite, except for one company that had only an internal SCM and no cloud stuff.

Re:GitHub: October 21 Incident Report by h1d on Tuesday October 23, 2018 @12:13PM (#18284658)
I'm not sure how hard it is to maintain some self hosted apps. It's just set up and forget. It doesn't randomly change interface or license unless you upgrade it as well.

I've used AWS for 10 years but for last 5 years, I've never seen it just go down randomly and even if it did, you have the room to redeploy with some clicks (assuming you have your data backed up regularly) instead of waiting for the uncontrollable.

You seem to take the ops work a bit overly.

Re:GitHub: October 21 Incident Report by marcosdumay on Monday October 22, 2018 @01:37PM (#18276618)
You know, the time required to host Gittea (or Gogs) is basically nil (you'll have to set your environment up for Github too). And by maintaining it, you mean making sure it's online and taking backups? Because I don't see how you can save any time on those by using Github.

Where you host it is of less relevance, because you can simply take your backup and server script and run them at any different provider any time you want.

Re:GitHub: October 21 Incident Report by limonkufu on Monday October 22, 2018 @03:30PM (#18277584)
I am currently leading the on premise hosting of a Gitlab instance and I can say this with ease: I have been spending 1 day of each week for ops work. Let it be helping people, database adjustments, admin stuff, hardware checks, etc.
Re:GitHub: October 21 Incident Report by ProAm on Monday October 22, 2018 @01:52PM (#18276744)
>It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over,

Welcome to the SAAS world. Where you offload institutional knowledge and hiring to a 3rd party and roll the dice.

Re:GitHub: October 21 Incident Report by nineteen999 on Monday October 22, 2018 @11:26PM (#18280511)
This is why we built and manage our own git servers internally. We run emergency services infrastructure so we can not afford to be disconnected from our version control. Ever.

Same reasons we don't run our application and supporting infrastructure in a cloud provider really - apart from the complexity of our particular network which has a heavy dependency on mobile and satellite links - we cannot guarantee our customer of our availability if we don't control the compute, storage, SAN fabric, and as much of the network infrastructure as is possible.

Re:GitHub: October 21 Incident Report by h1d on Tuesday October 23, 2018 @12:06PM (#18284577)
Why would state actor level attacker target for being a code hosing platform? At best they can temporarily suspend operations.