News for hackers. Stuff that matters.
Re:Investigate switching away from GitHub by makecheck on Thursday February 11, 2016 @04:16PM (#11083335)
A central-hub choice of implementation shouldn't be confused with a centralization requirement however. It is possible for distributed systems to still converge on things (indeed, entire papers have been written on the problem of distributed “agreement”), it's just that they must work in a very different way.

Indeed, we had this, in the form of NNTP and older news servers; those servers essentially exchanged copies of whatever was posted from around the Internet. If there were no news.ycombinator.com, there could still be a wide variety of servers that constitute “hacker news” around the world, that come to a general agreement on what the collection of posts should be. At any instant in time, not all servers will be 100% identical but for all practical purposes they would be.

Re:Investigate switching away from GitHub by jasode on Thursday February 11, 2016 @04:28PM (#11083411)
>A central-hub choice of implementation shouldn't be confused with a centralization requirement however.

>Indeed, we had this, in the form of NNTP and older news servers;

You're focusing on the technical aspects. The other forces are social and economics. Both of those lead to centralization.

Take your example of Usenet news groups. Yes, in a technical sense, it's decentralized but once you get economics/money involved, what happens? My ISP like Verizon doesn't carry all newsgroups (e.g. exclude newsgroups with large binaries.) Or another ISP may carry more newsgroups but the retention is 1 month instead of 1 year.

So the economics converges to an central entity like Deja News that emphasizes the archival of usenet posts. Then Google buys them (Google Groups). Google Groups is now "centralized" for finding very old usenet posts. Conceivably, any of us besides DejaNews could have archived the posts but we didn't spend the money on harddrives to store it whereas DejaNews did. Hence, we end up with a centralized archive.

When I had Verizon, their newsgroups feed was very poor. I therefore paid $10 extra to GigaNews because they had everything with multi-year retention. That's another "centralized" hub driven by economics.

Looking only at the usenet RFC technical specs for distributing news posts doesn't reveal how the economics makes centralization emerge.

As for github's centralization...

A lot of us want to host a public git repository. Hmmmm... if github didn't exist, what do we do? Pay for a VPS for $9/month plan and host git on that? What about extra bandwidth costs if it's a popular repo and lots of people are cloning it? That's a lot of ugly financial bookkeeping that we don't want to do since we'd rather just concentrate on programming. Wait -- there's an ABC company out there that lets me conveniently host a git repo there? For free? Sign me up!

That "ABC company" by way of economics and social dynamics has turned out to be Github and BitBucket. And since Github has been winning the mindshare, you get centralization. There is no irony in git-the-tool being distributed and github-the-focal-point being centralized.

Re:Investigate switching away from GitHub by mbrock on Thursday February 11, 2016 @05:42PM (#11083911)
I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function. There are lots of benefits to the GitHub model. They're a semi-trusted and well-funded entity who guarantee a highly available and fast synchronization. Sure, when it goes down everybody has a problem for a while... but with more distributed models you can get way more complicated failures, like "this object isn't seeded by a connected peer right now" or "this federated shard doesn't carry this repository" or whatever it may be.

I'm really interested in the future of distributed code collaboration. I just think it's not so trivial, and so it's very understandable that people have gravitated towards a semi-centralized model that already works.

Re:Investigate switching away from GitHub by u801e on Thursday February 11, 2016 @09:26PM (#11084917)
> I agree that NNTP was a nicely federated system, and just a couple of days ago I was pondering the use of NNTP to publish Git pushes. Git's support for signed commits and content addressed objects is a great start for distributed, trustworthy collaboration. I still can't envision exactly how it would all function.

Perhaps having something akin to a private NNTP server that requires authentication to post for each open source project, but is publicly viewable. These news servers could be configured to send IHAVE <article> to other news servers to increase visibility. There could be a newserver like gmane to aggregate the feeds from each of those private news servers.

If you could get an authentication system that could work across those news servers, then you could have a decent system for keeping up with and contributing to open source projects.

Re:Investigate switching away from GitHub by icedchai on Thursday February 11, 2016 @04:31PM (#11083431)
Sadly, most people have forgotten about Usenet... or never knew it existed. The typical web forum is vastly inferior to most news clients.
Re:Investigate switching away from GitHub by mbrock on Thursday February 11, 2016 @05:44PM (#11083926)
Well, Usenet had basically no way of doing moderation, right? That's a pretty big missing feature from the perspective of forum administrators.
Re:Investigate switching away from GitHub by ksherlock on Thursday February 11, 2016 @08:28PM (#11084727)
There are moderated groups. The NetNews Moderator's Handbook[1] has technical details (and more) but basically:

    'Moderated' means that all postings to the newsgroup go to a mail
    address (e.g. [email protected]) instead of appearing
    in the newsgroup directly. The postings are then forwarded via
    email to a moderator, or group of moderators, or even an automated
    program, who decides whether to actually inject the article into
    the newsgroup or to reject it as not meeting guidelines spelled
    out in the group's charter.

1: https://www.eyrie.org/~eagle/usefor/other/moderators-handboo...
Re:Investigate switching away from GitHub by quesera on Thursday February 11, 2016 @11:39PM (#11085354)
Notably, though, USENET moderation was managed entirely by the addition of a single, unverified, header to the message ("Approved: cuzisayso"). This allowed for simple distribution, but equally simple circumvention.

Proper moderation for anything important would require cryptographic signatures and a solid federated PKI. Maybe the world is ready, this time. :)

Re:Investigate switching away from GitHub by mbrock on Friday February 12, 2016 @05:02AM (#11086238)
I think the world is ready for someone to make an actual implementation and start using it for some actual conversation!

As long as you have an actual living conversation, then an open model of development can work... which means it doesn't have to be perfect from the start, and you don't need a beautiful landing page, because you want to attract the hackers and crazy people.

You also want to encourage lots of forks and independent implementations, because if it turns into some big mega-repository with some big bloated thing that includes Scala, Erlang, Babel, Bootstrap, and who knows what other ungodly monsters, then it's just going to become a boring monstrosity.

So you'd start with inviting a few people and then encourage them to use their own servers and clients. Make sure everyone understands that the protocols are in flux because you're trying to figure out together what will work.

The conceptual simplicity of Git would be an inspiration. There are different implementations of Git... but we could be even more obviously amenable to that, by using standardized messaging formats for example.

I think plain text messages, as on Usenet and mailing lists, are the best. You don't have to agree on a Markdown/whatever complex thing. Attachments if you want to send other documents. Hacker News developed consensus formatting for links and footnotes, and we seem to get on fine without inline images and tables and all the other fancy nonsense.

(Just some random thoughts...)

Re:Investigate switching away from GitHub by quesera on Saturday February 13, 2016 @01:13AM (#11092820)
I agree. It'd be great fun to build something incorporating everything we knew about distributed and federated systems in the days before their incompatibility with business models became an issue.

There's a lot of good stuff in XMPP, too. As well as some pre-discovered dead ends.

Re:Investigate switching away from GitHub by mbrock on Saturday February 13, 2016 @10:01AM (#11094158)
Yeah. From a hacker's perspective, a business model is just some way to keep a thing up and running. If it doesn't create a bunch of jobs at some office somewhere or lead to an IPO, well, that's just worldly affairs, right?

The distributed/federated stuff is still alive, maybe more than ever. Git is a fantastic example. I push and pull repositories between computers all the time. Often over ad-hoc WiFi, even.

Usenet and all that stuff is beautiful to reminisce about, but we should keep in mind also that it was largely sponsored by universities, corporations, militaries, and internet service providers...

IRC is a great inspiration too. Nowadays people look at it and think "ew, no Unicode" or "duh, it's not even JSON," but considering its resilience, its simplicity, its adoption, it's an enormous, staggering success.

These beautiful projects that started as tiny little seeds by clever obscure Scandinavian hackers...

It's just all about starting a little thing, getting your friends in on it, and keepin' it hella real.

Re:Investigate switching away from GitHub by quesera on Saturday February 13, 2016 @09:33PM (#11096886)
True, although keep in mind that while USENET's bandwidth was definitely sponsored, the technology (above the TCP layer) was built by hackers as a side project. At some point Larry Wall (e.g.) got a job just being himself, but for a long time he had unrelated professional responsibilities.

Since the bandwidth funding model has now bubbled down to the consumer marketplace, it seems like the conditions are right for a huge variety of services. Actually that was starting to happen before the web swallowed all other services. Don't get me wrong, the web was/is a great thing, but it severely dumbs down the range of protocol expression. The state of websockets in 2016 is a weak imitation of IRC or MUD capabilities more than 20 years ago (never mind adoption problems).

I guess that's a curse of being "good enough". The better has an uphill battle.

Re:Investigate switching away from GitHub by spc476 on Thursday February 11, 2016 @09:21PM (#11084901)
And there was one group, alt.hackers, that was moderated, but had no moderator, by design. It kept the signal-to-noise ratio pretty high.
Re:Investigate switching away from GitHub by icedchai on Friday February 12, 2016 @12:07AM (#11085440)
You could "kill file" a user, which meant you wouldn't see any of their posts, but other people still could.

What I like about news readers is how they track new threads / messages. I could pick up where I left off. It encouraged longer term, more elaborate discussions, as opposed to posts that fall off the first couple pages like most web forums.