News for hackers. Stuff that matters.
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:13PM (#22936269)
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here:

https://man.sr.ht/ops/

It's also the highest performance software forge by objective measures:

https://forgeperf.org/

Full disclosure: I am the founder of SourceHut.

Re:GitHub is degraded/down by sky_rw on Tuesday April 21, 2020 @12:24PM (#22936399)
It's unfair to Github to make the claim that your infra is more resilient and better maintained. Their load is orders of magnitude greater than yours. My driveway also doesn't have potholes it doesn't mean it's more resilient than the freeway.
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:29PM (#22936479)
I don't think so. I backed up the claim here:

https://news.ycombinator.com/item?id=22936448

SourceHut is at least 10x lighter weight and has a distributed, fault tolerant design which would allow you to continue being productive even in the event of a total outage of all SourceHut services.

Re:GitHub is degraded/down by erikpukinskis on Tuesday April 21, 2020 @12:27PM (#22936444)
Sidebar, I just want to say, you are one of the few people I’ve observed doing actual “modern” web development.

When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.

That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.

That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.

ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.

You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.

That’s true modernism.

So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.

Re:GitHub is degraded/down by SparkyMcUnicorn on Tuesday April 21, 2020 @01:28PM (#22937093)
Just wanted to interject that browsers (other than IE 11) have over 98% coverage for ES6 without transpiling.
Re:GitHub is degraded/down by cm523 on Tuesday April 21, 2020 @12:58PM (#22936780)
Care to expand on this some more? Perhaps you have some other examples of both good front-end and server side, modern, day web development?
Re:GitHub is degraded/down by bdcravens on Tuesday April 21, 2020 @12:23PM (#22936392)
I'm sure you've done a great job building up your infrastructure, but if you have the level of traffic Github has, what would your uptime be?
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:27PM (#22936448)
Who's to say? It's not GitHub scale, and even if everyone in this thread moved to SourceHut, it still wouldn't be GitHub scale, but it would be serving your needs just fine. I feel totally comfortable recommending SourceHut over GitHub as a service which can be expected to have better uptime and performance, because it is a fact - even if we operate at different scales.

And I believe sr.ht would beat out GitHub at their scale anyway. The services are an order of magnitude more lightweight. And the design is more fault tolerant: we use a distributed architecture, so one part of the system can go down without affecting anything else - as if GitHub's issues could go down without anything else being affected. And many of our tools are based on email, a global fault-tolerant system, which would allow you to get your work done more or less unaffected even if SourceHut was experiencing a total outage. We'd automatically get caught back up with what you were up to in the meanwhile once we're online, too.

I've spoken to GitHub engineers about some of the internal architectural design of GitHub, too, I'm confident that SourceHut's technical design beats out GitHub's in terms of scalability. And, despite already winning by a good margin, I'm still spending a lot of effort to push the envelope further on performance and scalability.

Re:GitHub is degraded/down by saagarjha on Tuesday April 21, 2020 @01:08PM (#22936899)
> Who's to say?

And then you go on to say it. I'm glad that SourceHut exists, and I like many of its principles, and it's probably better designed too, but walking into a thread where someone is having an outage and then claiming that you'd do much better is in poor taste no matter how good you are or how many of your services work offline.

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:17PM (#22936980)
I responded directly to someone who said they were considering alternatives, and wouldn't've otherwise.
Re:GitHub is degraded/down by saagarjha on Tuesday April 21, 2020 @02:39PM (#22937970)
Right, and I think it is great to bring up how your service can handle outages better than GitHub would due to it being decentralized. The part I have issue with is saying that you'd do better than GitHub about keeping your site up, pointing to the issue that they are in the middle of resolving–that just seems like kicking them while they're down, especially since you haven't actually shown that you can do better. (Yes, you have good uptime in the past, but I don't see what's stopping the power going out to some of your servers, or you pushing a bug into production, or any number of other things that shouldn't go wrong but often do, especially as the number of users increases.)
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @03:41PM (#22938589)
>what's stopping the power going out to some of your servers

Redundant power supplies

>pushing a bug into production

Nothing, but again, SourceHut is demonstrably better in this regard: because it's distributed, a bug in production would only affect a small subset of our system, and the system knows how to repair itself once the bug is fixed.

And I don't think I need to apologise for kicking Golaith while he's down. Someone said they want alternatives, so I pitched mine with specific details of how it's better in this situation, and that doesn't seem wrong to me. I would invite my competitors to do the same to me. We should be fostering a culture of reliability and good engineering - and if I didn't hold my competitors accountable, who will? "Here's an alternative" has more teeth than "I wish this was better."

[deleted]
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:30PM (#22937112)
I'm referring to the commit to which I initially replied:

https://news.ycombinator.com/item?id=22935985

"...is it time to move away from Github?"

Re:GitHub is degraded/down by bdcravens on Tuesday April 21, 2020 @01:39PM (#22937252)
Yeah, I reread your comment, but you responded before I deleted apparently :-) My mistake.
Re:GitHub is degraded/down by bdcravens on Tuesday April 21, 2020 @01:23PM (#22937040)
> SourceHut over GitHub as a service which can be expected to have better uptime and performance, because it is a fact

Most of us could throw any of the open source solutions on a $20 Linode instance and probably have excellent uptime. How many active repos do you host, and on how many servers?

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:28PM (#22937087)
About 18K git & hg repositories, for about 13.5K users. We also run about 5,000 CI jobs per week, including for some large projects like Nim and Zig, Neovim, OpenSMTPD, etc. We have 10 dedicated servers at the moment. And I didn't throw an open source solution on these servers - I built these open source services from the ground up.
Re:GitHub is degraded/down by bdcravens on Tuesday April 21, 2020 @01:39PM (#22937240)
So you're comparing your scalability with a company with over 40m users and 100m repos.

Can you talk about the geographic distribution of your 10 servers?

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:43PM (#22937318)
I would like to remind you of my earlier point:

SourceHut is not the same scale as GitHub. This does not change the fact that SourceHut is faster and more reliable. We have an advantage - fewer users and repos - but still, that doesn't change the fact that we're faster and more reliable.

This has been objectively demonstrated as a numerical fact:

https://forgeperf.org

And yes, 9 of those servers are in Philadelphia (the other is in San Franscisco, but it's for backups, not distribution). That doesn't change the fact that, despite being more distant from many users, our pages load faster. In this respect, we have a disadvantage from GitHub, but we're still faster.

GitHub and Sourcehut are working at different scales. That doesn't change the fact that SourceHut is faster.

Re:GitHub is degraded/down by bdcravens on Tuesday April 21, 2020 @01:50PM (#22937382)
I was considering your claim:

> we use a distributed architecture

> SourceHut is faster

I wasn't questioning that some of the web features are fast. I'm sure when Github was 10 servers their pages were fast too. I suspect if I threw Gitlab on a 9-server cluster on AWS they'd also be quick.

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:52PM (#22937416)
Not geographically distributed, but distributed in the sense that different responsibilities of the overall application are distributed among different servers, which can fail independently without affecting the rest. Additionally, the mail system on which many parts of SourceHut relies is distributed in the geographical sense, among the hundreds of thousands of mail servers around the world which have standard and 50-year-battle-tested queueing and redelivery mechanisms built in.

And yes, throwing GitLab on a 9 server cluster on AWS might be fast. But, I'm ready to bet you that SourceHut will be faster than it still, and I have a ready-to-roll performance test suite to prove it. And I know that SourceHut is faster than GitLab.com and GitHub.com, and every other major host, and you don't have to go through the trouble of provisioning your own servers to take advantage of SourceHut's superior performance.

Re:GitHub is degraded/down by fortuity on Tuesday April 21, 2020 @03:32PM (#22938511)
> This has been objectively demonstrated as a numerical fact:

While your tests are indeed objective, I don't think they're very useful. For example, why does your performance test ignore caching?

GitHub's summary page loads 27KiB of data for me unauthenticated, which is about 6% of the 452KiB you're displaying in your first table. The vast majority of developers who browse GitHub will not be loading 452KiB of static assets every single page load.

Anecdotally, GitHub's "pjax" navigation feels about as fast as SourceHut on my aging hardware.

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @04:37PM (#22939077)
Even with caching, SourceHut is a lot smaller than that. SourceHut benefits from caching, too - the repo summary page comes from 2 requests and 29.5K to 1 request and 5.7K with a warm cache. And in many cases, the cache isn't the bottleneck, either - dig into the Lighthouse results for specific pages to see a more detailed breakdown.
Re:GitHub is degraded/down by mwcampbell on Tuesday April 21, 2020 @12:31PM (#22936506)
Thanks for being so transparent about your operations.

Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?

Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:33PM (#22936527)
No, I provision them manually. Being based on Alpine Linux makes this less time-consuming and more deterministic. At some point I might invest in something completely automated, but right now the manual approach is simpler - and if it's not broke, don't fix it.
Re:GitHub is degraded/down by mwcampbell on Tuesday April 21, 2020 @12:45PM (#22936645)
Ah, OK. Also, have you written anywhere about why you chose to use colocation rather than VPS (or "cloud") hosting, or leased dedicated hosting for the CI system? If you could use someone else's hardware rather than having to select, buy, and set up your own, then at least in theory, you could spend more time on other things. But I'm sure you have your reasons for making the choice that you did. I'm just curious about what those reasons are, if you're inclined to share.
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:56PM (#22936761)
There are lots of reasons, but the most obvious one is cost. All of SourceHut's servers are purpose-built for a particular role, and their hardware is tuned to that. The server that git.sr.ht runs on is pretty beefy- it cost me $5.5K to build. I paid that once and now the server belongs to us forever. I ran the same specs through the AWS price estimator, and it would have cost ten grand per month.
Re:GitHub is degraded/down by maple3142 on Tuesday April 21, 2020 @01:38PM (#22937233)
A little bit off topic. I just wondering it support git write access over https? Not just read only.
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @01:40PM (#22937254)
No, write access is only supported over SSH, for security reasons. SSH key authentication is stronger than password authentication, and git.sr.ht doesn't have access to your password hash to check anyway.
Re:GitHub is degraded/down by mrlala on Tuesday April 21, 2020 @12:27PM (#22936440)
Yeah I'm sure there are no possible scaling issues between your service (how many people use it / repos acitve) vs github or gitlab....
Re:GitHub is degraded/down by salimmadjd on Tuesday April 21, 2020 @12:25PM (#22936422)
your $2/month pricing. Is it $2/person/month?
Re:GitHub is degraded/down by ddevault on Tuesday April 21, 2020 @12:28PM (#22936454)
Yes.