News for hackers. Stuff that matters.
Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @04:33AM (#4676648)
0mq doesn't have a client/server relationship. http://www.zeromq.org/
Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @04:42AM (#4676656)
I believe I looked at that as well. It was six months ago, so forgive me if my memory is a bit off.

Looking at the intro, it looks like I have to hardcode the IPs and ports of all of the machines I want to publish or subscribe to. Is that correct? If so, that's a little fragile for my tastes. I like "upgrading servers" by standing up new servers, testing they work, and then changing the floating IP associated with the DNS to point to them instead.

And I can hear you say "Aha! Floating IP! Just use that!" but for most cloud providers (I believe), that is billed as external bandwidth, which is not free like internal requests are. Finally, if I want to stand up more servers to scale horizontally, I'd still have to modify my code and redeploy it to all my servers. I think. Unless I'm missing something about how 0mq works. Which I totally could be.

On the whole, learning what I needed to learn to make Pastry happen made me a better programmer, too. So if for no other reason than that, I'm glad I did it.

Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @05:04AM (#4676679)
No more so than with TCP sockets you have to "hard code" the IP connected to.

0mq is low level, it's not going to do routing like Pastry, but you can get away with having one fixed location for a "lookup server" that keeps track of everything else's location (something like the Doozer project, also in Go https://github.com/ha/doozerd ).

Also, 0mq only needs to know the IP of the publisher, for pub-sub.

Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @05:15AM (#4676690)
Very cool. Where were you six months ago? ;)

I was familiar with doozer, but doozer wasn't(/isn't) being actively maintained and it doesn't compile against the latest version of Go, so I'd have to bring it up to speed before I could use it anyways. Not saying it would be harder, but between that and writing the 0mq library (pretty sure one does not exist in Go yet), I'd estimate the work would be more or less equivalent and yield just about the same stability. Seat of the pants guess, but it makes me feel better, at least.

Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @05:34AM (#4676715)
Heh, thanks.

The mailing list suggests http://github.com/4ad/doozer and http://github.com/4ad/doozerd are being maintained as forks of the original. Or there's Zookeeper. Or brew your own central lookup server using 0mq that does the same job of "set a key, get a key, notify subscribers when it changes". It's a single point of failure, but since the system will run fine without it (only lacking topology updates) it's not a hugely problematic one.

Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @06:02AM (#4676741)
This reply link just appeared for me. Not cool -_-

The lookup server is a single point of failure. And while it's not a hugely problematic one, it also is a dedicated machine whose sole purpose is keeping the other machines running. Which feels wrong to me.

It probably wasn't the best business decision to invest time in this, and I won't even argue this is the best technical solution. But it's the only technical solution that didn't make me feel like I was working around limitations; things just worked they way they were supposed to. The API servers received an event, they told the WebSockets servers about it. It felt very conceptually pure to me. I'm a sucker for that.

Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @06:21AM (#4676767)
To get the reply on a deep comment, click "link" first to get the comment as a single page, and "reply" will be there.

Doozer is only slightly a SPF, it has high availability by having multi-master replication. It's also not doing a lot of communicating, or a lot of CPU work, and it keeps its data in RAM, so it may be OK for it to live on a non-dedicated machine.

Sorry if I come across as criticizing your admittedly cool creation. It was just that you said there weren't alternatives, and I knew of one.

Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @06:25AM (#4676770)
No, I definitely appreciate the conversation, because I looked for forever for a pre-built solution to this, and couldn't figure out how nobody had needed a solution before now. So I'm glad there are solutions, and I'm not just crazy.
Re:Pastry: A distributed hash table in Go by alexchamberlain on Saturday October 20, 2012 @05:58AM (#4676733)
Isn't the lookup server a SPoF?
Re:Pastry: A distributed hash table in Go by brian_cloutier on Saturday October 20, 2012 @04:42AM (#4676655)
What does it have?
Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @04:53AM (#4676671)
Peer-to-peer connections analogous to TCP. In the case of pub/sub all subscribers connect to the publisher. http://zguide2.zeromq.org/page:all
Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @05:03AM (#4676678)
I guess I was wrong about hard-coding the IPs. Maybe?

It doesn't have many-to-many baked in, though. That I can see, at least.

It looks like I may have been able to make this work, but the setup would be a little complex.

... because a distributed hash table is not complex at all. -_-

Re:Pastry: A distributed hash table in Go by JulianMorrison on Saturday October 20, 2012 @05:26AM (#4676705)
It doesn't have many-to-many but in my experience most things aren't true many-to-many. More like, there are a handful of publishers in the system and each has a handful of subscribers.

Suppose your architecture is API -> Web sockets and you want to make it highly scalable, then what I'd do is:

1. Many API -> few relays (use doozerd to keep up to date as to where the relays are and select one randomly, use ZMQ_ROUTER socket type to enqueue API->relay messages).

2. Few Relay -> many web socket (again using doozerd for the web socket nodes to find the relay, and using ZMQ_PUB on the relay and ZMQ_SUB on the web socket nodes).

Re:Pastry: A distributed hash table in Go by paddyforan on Saturday October 20, 2012 @06:03AM (#4676742)
Sounds like it would work, and have great availability and scalability. It just, conceptually, doesn't mesh with how I think about the system. An absolutely aesthetic distinction, I know.