News for hackers. Stuff that matters.
Re:What Twitter could have been by SwellJoe on Sunday July 01, 2012 @03:26PM (#4185831)
A "firehose" API for the net would be awesome, but it probably needs to be Open Source, in order to avoid being driven towards monetization. Of course, without the drive to make money, Twitter may have never taken off quite the way it did, and may have never had the money to spend on SMS and other expenses they were dealing with in the beginning to create that firehose of data.

The Open Source twitter clones (like identi.ca) have modeled what Twitter became, mostly, rather than what Twitter could have been...so, as far as I know, there isn't much out there that answers this description. I'd started working on something I was calling SYSRSS years ago, which was to be a realtime systems human and machine readable data stream protocol and implementation, but never got a minimum viable product out the door, as I could never quite figure out the right niche for such a thing. I think it may have been too specific, though building on top of an existing popular data format and protocol was probably a valid choice.

In short, I agree that Twitter could have been more awesome than what it is. But, I don't know if I could have made it better, or that having the API guys win would have succeeded. I haven't done any research into that field lately...I wonder if there are any new developments in the Open Source world for this kind of thing?

Re:What Twitter could have been by cloudmike on Sunday July 01, 2012 @04:18PM (#4186003)
You should check out Simperium (disclaimer: I'm the co-founder and Twitter's former platform lead is one of our advisors/investors). While it's not immediately apparent yet, we're pushing in this direction: it's easy to create a firehose for your app, and we're working on ways for apps to easily interconnect with each other in a less file-centric, more API-centric way.

The niche we chose to start with is device syncing, based on our experience building Simplenote (which is to Evernote as Twitter is to Facebook). We know there's a need for something more flexible and powerful than iCloud in this regard, so that's what we focused on initially. But we believe a good solution for moving data between devices is also a good solution for moving data between people and 3rd-party apps.

Simplenote's 3rd-party ecosystem looks similar to Twitter's in the earlier days. We're taking the path they didn't take. Rather than shoring up Simplenote vertically, we're focused on extending our backend data layer horizontally and generalizing it for a variety of apps and services (though specifically not as an all-encompassing "backend-as-a-service"). We'll be announcing our open source strategy soon. This is all really hard though, both technically and as a business.

Re:What Twitter could have been by paulsutter on Sunday July 01, 2012 @04:40PM (#4186059)
Can you think of examples of any open source standards that emerged from pasting on an open source strategy later? I ask because I can't think of any, but I may be wrong.

I'm contemplating a realtime data sharing system now, but it's hard for me to consider using a commercial platform from a startup.

If your API was already open source, with a real open source community that spans several organizations already in place, I'd be a lot more eager to use your service.

Basically, an open source api is only appealing to me if it seems impossible for it to be hijacked by one sponsor, no matter how well meaning. Who knows, you could get hit by a bus and wake up as your own evil twin.

Love to hear your thoughts

Re:What Twitter could have been by cloudmike on Sunday July 01, 2012 @04:51PM (#4186085)
We only launched Simperium a few weeks ago. While it's production-ready (powering Simplenote), it still has a beta label for the simple reason that we haven't announced pricing or our open source strategy. I agree, "pasting on" an open source strategy in a reactionary way doesn't have a great history of success, but similarly, I think it's also hard to build very ambitious open source solutions that are entirely open from the moment of conception. Not impossible, but in any case the path we chose was to build a first version behind closed doors according to a strong vision based on experience and talking to fellow app developers.

Having done that, we're excited to start opening it up and we don't view this as pasting on a strategy. We've been proactively considering the implications for our technology and business from the beginning. I think a great, open circulating system for the internet can be as significant as an operating system like Linux.

Re:What Twitter could have been by paulsutter on Sunday July 01, 2012 @05:10PM (#4186151)
> I think it's also hard to build very ambitious open source solutions that are entirely open from the moment of conception

Yes it's hard to build an open source standard using an open development model. On the other hand some people think its impossible to build an open source standard without an open development model.

Open source advocates argue that the very barrier to something becoming an open standard is your wish to own a strong consistent vision. They also argue that having a single host company that has a head start over others will discourage others from jumping on board.

I sincerely hope that such an approach could lead to a real open standard. I can't think of a case that has, though I do hope to be proven wrong.

Cool project, for sure. I bet you can succeed as a commercial platform, you're solving some hard problems and will make life easier for lots of developers. Would be cool if you can drive an open standard too.

Re:What Twitter could have been by cloudmike on Sunday July 01, 2012 @05:27PM (#4186219)
Thanks! And I appreciate your insight. You must be rather battle-hardened from your experience over the years, and I know there are others who share your hesitance to use a system like this. In response to your question about examples, Linux and Android are examples of software that, as far as I can tell, only really opened up after conception and a good deal of development.

http://en.wikipedia.org/wiki/Linux#MINIX

http://en.wikipedia.org/wiki/Android_(operating_system)#Foun...

Re:What Twitter could have been by SwellJoe on Sunday July 01, 2012 @10:06PM (#4186918)
I don't think Linux can be compared in any way, because it was never a commercial venture, and Linus released the first version in a barely working skeletal form, and proceeded from there.

If you actually meant MINIX, which the link goes to, it's also a poor example because it is pretty much a failure...it lost out to the always-open Linux, in such a catastrophic manner that most people, even most Linux users, don't even know it exists or that Linux was a reaction to it. If you plan to be the "MINIX of realtime data streams", I don't think I'll be developing for your platform.

Android is possibly a reasonable example, as I guess it did happen as a proprietary venture that went Open, but it's Open Source life happened under the oversight of Google, arguably the world's most important technology company. You don't have that kind of clout or that kind of market penetration. No one worried that Google would disappear next week, leaving a bunch of development spending un-recouped.

Finally, there is a pretty hard rule of the Internet, which is that infrastructure will be Open Source...and if it isn't today, it will be soon. HTTP, DNS, SMTP, SSL, video (HTML5, which replaced the proprietary Flash). There are still a few examples of non-Open services, and Web 2.0 (or whatever), was a valiant attempt to retake the web for proprietary interests; and the jury is out on whether Open Source and standards will eventually prevail. So, chat is back to being often proprietary, video and voice conferencing never had a viable Open Source option so it remains proprietary with competing standards, etc. But, I think for you to convince other companies to build on what you make, you'll have to either become big fast (like Twitter fast), or start Open. Or, you'll build just another also ran.

Of course, if you start Open Source, you'll never make a bajillion dollars on the software or service itself. So, you get to choose your priorities, and maybe shooting for the big money is more important than a better shot at building a standard that many people and companies use.

Re:What Twitter could have been by marshray on Sunday July 01, 2012 @08:25PM (#4186640)
I think there a small contradiction in terms here, perhaps you've recognized it already.

A network service has a definition in in terms of the protocol used to talk to it. It could be an "open" protocol standard, in the sense that it's well documented and not patented. Even if it's not well documented, network protocols are not that difficult to analyze.

"Open source" is talking about source code to a program. Such a program might be an embodiment of a protocol endpoint.

The problem that Twitter has is that they have a trivial API and it's very easy for anyone to create a client for it. They're trying to lock this down cryptographically using client certs, but as long as there is some client on the users' devices that can talk to Twitter, the private key has to be available to all the users. Open source client endpoints have to publish their private key in the source. What they're trying to do is technically impossible.

So Twitter has chosen the worst of possible worlds here, they have a de-facto open API and must resort to the threat of lawsuits to prevent their users benefiting from the features provided by the open marketplace of client apps.

I would try to find a way not to end up like Twitter.

Re:What Twitter could have been by kalleboo on Sunday July 01, 2012 @05:21PM (#4186195)
> Can you think of examples of any open source standards that emerged from pasting on an open source strategy later? I ask because I can't think of any, but I may be wrong.

Does Java count, or did the recent Oracle lawsuits kill that one?

Re:What Twitter could have been by melvinmt on Sunday July 01, 2012 @06:39PM (#4186392)
> Simplenote (which is to Evernote as Twitter is to Facebook)

Are you looking for the word 'competitor'? I don't get these X is to Y as A is to B references..

Re:What Twitter could have been by politician on Sunday July 01, 2012 @04:29PM (#4186034)
Telehash is the Open Source "firehose API" you seek. The proposal describes a DHT that routes JSON. However, it seems like I've been waiting for it for years. Maybe it's the obscure telephone lingo that the API is described in terms of that's discouraging would-be implementers, but it's nothing that a good old facade can't fix.

http://telehash.org/

Re:What Twitter could have been by jamii on Sunday July 01, 2012 @10:13PM (#4186939)
> However, it seems like I've been waiting for it for years.

https://github.com/jamii/erl-telehash is a reasonably complete implementation (it's only missing rate limiting and auth).

After working on erl-telehash I have some doubts about the protocol details (although the overall idea still appears sound). I described how I would modify it at https://gist.github.com/1910754 but I'm not moving forwards with it at the moment (working full-time for now).

Re:What Twitter could have been by jchrisa on Sunday July 01, 2012 @07:32PM (#4186496)
I like to think of Telehash as the answer for p2p routing. It's powerful and there's working implementations.

The other piece of the puzzle is REST in the UDP context, for that, CoAP looks really well put together: http://tools.ietf.org/html/draft-ietf-core-coap-10

I have a weekend project going to blend the two and make magic, but I haven't put enough time into it yet.

Re:What Twitter could have been by jeremie on Sunday July 01, 2012 @11:00PM (#4187088)
Is there decent CoAP implementations? That's unfortunately one of the big stumbling blocks with universal p2p transport yet :/
Re:What Twitter could have been by demachina on Sunday July 01, 2012 @08:36PM (#4186666)
Telehash is radically decentralized. How do you get a firehose out of that? It would be more like a point to point garden hose wouldnt it? I would like to be proved wrong, but I think Twitter/centralized and Telehash/decentralized will end up looking and acting like two very different services.

If the Twitter as an API group had won out the two best outcomes I can think of are either a buy out from Google, or a move to a foundation like Wikipedia. Google could afford to carry it, like it does Youtube, just so they could own a huge firehose of information and use it elsewhere in search ads, without trashing the stream with ads.

Needless to say founders looking to get rich wouldn't go for the low returns of the foundation model but it would have been the best possible outcome for the well being of the future Internet having and open, censorship resistant message bus and API, assuming you could get enough donors to support it.

Re:What Twitter could have been by jeremie on Sunday July 01, 2012 @10:57PM (#4187083)
With a telehash style distributed messaging bus you'd need to have clients opt in to centralized reporting and analytics providers, like the library of congress, internet archive, even commercial firehose companies like gnip, etc.
Re:What Twitter could have been by wslh on Sunday July 01, 2012 @09:54PM (#4186879)
Also analytics will be very difficult in a P2P architecture. That is the power of twitter: you only need to analyze the centralized firehose.
Re:What Twitter could have been by chipsy on Sunday July 01, 2012 @05:05PM (#4186135)
One of the most ambitious projects I've seen in this vein is psyced ( http://www.psyced.org/ ) which starts from a redesign of IRC and has gone on to accumulate a kitchen-sink's worth of protocol compatibility(from the front page: Jabberâ„¢/XMPP S2S, IRC, TELNET, HTTP, SMTP, OAuth, XML, RSS; limited/experimental code for XMPP C2S, Java Applets, Status.Net, Twitter API, WAP, NNTP.) The core protocol is a gorgeous "Right Thing" design which has had many years to mature, and attempts to cover all venues: IM, chat room, microblogging, mail, profiles...

However, like with all of these open-source efforts, traction as a network seems to be missing.

Re:What Twitter could have been by J_Dalton on Monday July 02, 2012 @05:17AM (#4188063)
actually the core PSYC folks have moved on to develop http://secushare.org which is a P2P social beast "done right" with technologies and syntaxes much faster than XMPP or JSON. secushare is getting plenty of attention in the european hacker community and projects like freedombox. it's the missing link basically.