News for hackers. Stuff that matters.

A comment on Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines"). Parent comment

Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by pg on Saturday April 28, 2007 @11:36AM (#17705)
For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging in when you're not logged in and you submit a link through the bookmarklet) is part of the state of the closure; in that case it's a bit like a continuation.

We keep only the most recently generated 20,000 closures. When you click on a link that says it has expired, that means either it has been purged from the closure table, or that we've restarted the server.

[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by ralph on Sunday April 29, 2007 @04:43AM (#17838)
What extra data is in the closure for "reply" that means it isn't a simple URL? Is it just what to do after the reply is complete, i.e. what post to return to viewing? Is there a reason this isn't taken from the Referer instead?

I guess this technique doesn't scale too well unless the hash table is either shared across servers or a user is kept on one server, which then causes problems of its own.

I'm guessing 20,000 closures is plenty and I've been bitten by server restarts. Out of curiosity, do you keep track of the minimum age of the 20,000th hash entry prior to deleting to make room for another? If that's something huge, it would confirm server restarts are the normal reason the "unknown fnid" message is seen.

[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by avibryant on Monday April 30, 2007 @11:32AM (#18129)
What particular problems are caused by the user being kept on one server?
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by ralph on Friday May 04, 2007 @01:01PM (#19432)
The redundancy of having five servers isn't helping if I'm always served by server #3, whereas an alternate method where any one of the five can answer my current request allows a machine to fail, or be taken down for upgrading, and the load on the other four just increases by 25% each.
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by papersmith on Saturday April 28, 2007 @04:04PM (#17747)
"in that case it's a bit like a continuation."

Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by shiro on Saturday April 28, 2007 @07:57PM (#17785)
From my experience of continuation-based web app, what you're dealing with is not a full continuation captured by call/cc, but rather a delimited (partial) continuation, which captures the continuation of your application logic but not the state of underlying protocols. You can implement delimited continuations on top of call/cc, and I did based on Gasbichler&Sperber paper (ICFP02), but in most cases explicit CPS wrapped in some macros are just as well worked.
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by pg on Saturday April 28, 2007 @05:01PM (#17765)
It does, and so does Arc, but I don't think we've needed them in anything we've written so far. News.YC is a pretty simple application.
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by damir on Saturday April 28, 2007 @08:00PM (#17787)
Can we expect to see Arc go public in the near future?
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by malkia on Saturday April 28, 2007 @01:45PM (#17719)
Isn't there a risk, that someone can (accidentally or not) enter hash-code of someone else's "closure"?

[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by mdakin on Saturday April 28, 2007 @02:05PM (#17722)
One could avoid that problem by keeping track of the owner of each closure and ensuring the user is properly authenticated before calling the closure.
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by pg on Saturday April 28, 2007 @03:19PM (#17735)
The code explicitly protects against that.
[ Parent | on HN ]
Re:Is this how news.ycombinator.com runs? (Ctrl+F for "Closu... by papersmith on Saturday April 28, 2007 @03:59PM (#17744)
If access to the closures are session-specific, you'll have to hijack the session to do that.
[ Parent | on HN ]